ESF in the Programming Period 20072013 : The Role of the Evaluation in the Member States of the E'U'

1
ESF in the Programming Period 2007-2013 The
Role of the Evaluation in the Member States of
the E.U. Exchange of Good Practices Athens,
November 21 2008
Requirements (and Pre-requirements) for a quality
impact evaluation. Problems and solutions emerged
in the Italian experienceof the 2000-2006 ESF
program evaluation

• Paolo Severati
• ISFOL-National ESF Evaluation Unit

2
Why so few impact evaluations?

• Nowadays (many) researchers are able to carry out
  counterfactual analysis and different statistical
  packages are affordable or freely downloadable
  from the web.
• Yet, especially in countries like Italy, where
  impact evaluation started to develop under the
  impulse of the Structural Funds Regulations,
  rigorous evaluations regarding the effects of
  public policies are very rare and limited to
  small interventions in small provinces.
• Where is the problem? Or, what are the major
  difficulties in order to carry out good impact
  evaluations? Why so few impact evaluations?

3
Real world evaluation constraints (Bamberger
2006)

• Budget constraints (they limit the number of
  human resources you can hire or use, the
  interviews you can make, the building up of
  statistical and information systems, ..)
• Time constraints (when the evaluation begins and
  ends)
• Data constraints (quality and accessibility of
  data)
• To these type of constraints, I would like to
  add a new one
• Human resources constraints (number, quality,
  culture and reference values, type of contracts)
• N.B. in my intervention, I will leave aside
  political constraints and will assume that policy
  makers do like evaluation and are able to set up
  a clear demand for evaluation or more likely are
  able to cooperate with evaluators in defining
  what could be the evaluation demand

4
ESF program evaluation the Italian experience
between 2000 and 2006

• Budget constraints financial resources devoted
  to evaluation (and the institution of evaluation
  teams together with monitoring and evaluation
  committees)
• Time constraints very strict constraints. Middle
  term evaluation (2003) and final evaluation
  (2005)
• Data constraints much progress in respect to the
  1994-1999 ESF period but during all over the
  period still work in progress. Financial
  resources devoted to the building up of regional
  and national monitoring systems and for
  retrospective surveys on beneficiaries
• Human resources constraints at the national
  level, highly skilled persons working in public
  administration (Isfol, public research centre)
  having short term contracts or contratti di
  collaborazione a progetto at the regional level
  generally persons working as consultants in
  private evaluation societies or as short term
  employees in regional public evaluation groups

5
Impact evaluation, a very difficult task (1)

• Identification problems proper identification of
  the effects on some variables (micro or macro).
  Caveats
• Post hoc ergo non propter hoc not all the things
  registered in the statistics are caused by the
  program these things simply happened (and cannot
  be considered as intended or even unintended
  effects of the program)
• With or without defining a control group does
  not assure a correct identification of the
  effects, because treated and not treated
  individuals could have many (observable and not
  observable) differences

6
Impact evaluation, a very difficult task (2)

• Measuring problems
• Pre-post information baseline and end line data
  are necessary but are not good measures of the
  impact. If you want to evaluate the impact of a
  program you need a model (an econometric model).
  Indicators are useful instruments but cannot be
  conceived as substitutes for models.
• In order to measure the impact you need adequate
  control groups (and remember, if its difficult
  to obtain good information on treated, its even
  more difficult to get information on untreated or
  on treated by other similar programs).
• There are a lot of methods (randomization,
  regression, matching, ) defined in literature to
  estimate the impact of a program. Each one has
  points of strength and points of weakness and
  can be judged as suitable ones only with
  reference to the specific evaluation problem you
  have. Anyway, each method relies on strong
  assumptions this assumptions generally are not
  testable.

7
A very difficult task but a lot of literature

• The design and the strategy of evaluation require
  a lot of attention and propose many analytical
  problems. For sure the choice of appropriate
  control groups, the time span considered in
  measuring the effect of the policy or of the
  program, the choice of sound methods in order to
  treat selection and self-selection problems.
  There is a lot of academic literature on these
  themes (for an introduction see, for example,
  World Banks Nonie Manual and EU Working Paper
  n.6) . I dont want to add more. Instead I want
  to emphasize some neglected and constraining
  problems that any evaluator faces when he/she
  shifts from theory to the real making of
  evaluation

8
Good impact evaluation a step backward (1)

• I will start by two statements
• Good evaluation needs good monitoring (an obvious
  statement, you may say). Monitoring addresses a
  lot of statistical and more broadly logic and
  scientific problems (not so obvious, see Amati et
  al. 2007). Monitoring cannot be exhaustive and
  self-sufficient. If you want good and affordable
  information on programs you need jointly
  qualitative (interviews to privileged preferred
  witnesses, for example) and quantitative
  approaches (indicators, statistical
  classifications, robust variable categorizations,
  and so on) .
• Good evaluations require good surveys (another
  obvious statement, you may say again)
  retrospective surveys and panel surveys. Many
  countries, Italy particularly, has a strong
  tradition in this field but is relatively young
  on survey expressly conceived for evaluation
  scopes.

9
Good impact evaluation a step backward (2)

• I would like to take your attention on two
  observations
• Experts in monitoring, experts in statistical
  surveys and experts in evaluation rarely
  cooperate.
• Monitoring, surveys, evaluation time consuming
  activities. Very expensive activities, especially
  in countries, like Italy, not used to do it.

10
Data are not freely disposable in nature

• Impact evaluation requires good data but data do
  not exist in nature. Data are the product of a
  long, exhausting process which involves many
  people.
• So, the collection, the organisation and the
  preservation of data are costly.
• However, few people are really aware that
  building up complex information system is a major
  obstacle.
• Im not referring to informatics or to
  statistical questions. There is a lot of
  literature about it.

11
The problem broad consensus on the importance of
(monitoring and) evaluation

• Both monitoring and evaluation require a strong
  political and institutional consensus regarding
  their scope and objectives, not only on the part
  of policy-makers but also on the part of other
  important and often neglected actors.
• Broad monitoring and evaluation characteristics
  should be known at administrative operational
  levels in order to gather good information on
  beneficiaries or on the way interventions are
  implemented by agencies. Without this consensus,
  for example, application forms remain pieces of
  paper instead of being recorded files treatable
  with statistical packages and, information will
  be incomplete and vague if thought to be used for
  formal fulfilment only. As a consequence,
  difficulties will be encountered, non only in
  implementing evaluation, but first of all in
  monitoring.

12
Common knowledge (culture) and shared values

• In short, a good evaluation requires
  sophisticated methods but sophisticated methods
  cannot be used without good and complete data
  entry processes.
• Good data will be available only when everyone
  knows that they will be used for research (by
  honest, capable and brave researchers) in order
  to understand if public policies work or not.
• Each one should consider himself/herself as an
  important part of very complex evaluation process.

13
Italy a wonderful country with a terrible
governance

• I want to briefly describe how these problems
  were tackled in
• Italy in the context of the European Social Fund
  (ESF).
• Italy has, as you probably know, a very complex
  governance system (20 Regions and 109 Provinces),
  especially with regard to active labour market
  policies, and in particular, vocational training.
  In these matters Regional and Provincial
  Authorities often overlap each other and enter in
  conflict.
• Italy is made up of 21 ESF Managing Authorities
  (19 Regions and 2 Autonomous Provinces), each one
  very different from the other. Some regions
  reached and exceeded Lisbon targets before the
  set date, some others are still far from these
  targets (so, on average, Italy is now under the
  Lisbon targets). Local monitoring systems are
  very heterogeneous and evaluation culture is not
  evenly developed.

14
The Italian evaluation system

• Our evaluation experience is based on two
  different approaches in the first one the centre
  promotes and carries out the national monitoring
  system, beneficiary surveys and impact
  evaluations, in the second one the centre
  stimulates the regional authorities in building
  up the same activities by themselves under a
  clear methodological guidance in order to favour
  capacity building

15
Some remarks on the Italian experience

• In these last two years our principal aims have
  been the following two
• We have been stressing the major difficulties in
  obtaining data for the construction of internal
  and external control groups and what could be the
  risks and the advantages in using administrative
  data and in integrating different archives.
• We have been supplying a lot of instruments for
  simplifying and harmonising the statistical data
  collection processes (entry application forms
  data transmission protocol placement
  questionnaires ORFEO-Fields of Training KEY
  table, Fields of Training search engine ).
• We are confident that our technical-scientific
  consultancy to managing authorities on these
  subjects will repay us (and all the other
  independent evaluators) when we (or they) will
  start evaluation. At that moment, we (they) will
  have the information we (they) need in the way
  you (they) need.

16
So what? Why so few impact evaluation?

• Up to now adequate conditions failed. In short
• Appropriate data sets were lacking, so were
  capable researchers.
• Evaluation logically and temporally follows
  monitoring and beneficiary surveys, but all the
  activities should be planned as a whole at the
  same time, otherwise you will be obliged to adapt
  what you have and you will lack what you need.
  For too much time many people thought of
  evaluation, monitoring and surveys as single and
  distant universes.
• Financial and human resources devoted to
  evaluation are provisional. EU Members States,
  Italy in particular, should invest more on
  evaluation. I hope evaluation (conceived as a
  function) will become, in my country (my
  beautiful and terrible country), permanent and
  will not be any longer thought as a mere
  accomplishment to European obligations by any
  actor.
• Probably my initial hypothesis was wrong. Are we
  really sure that policy makers are glad to be
  (monitored and) evaluated on the work done?

17
Desperate conclusion

• What is a nice girl like you (impact evaluation)
  doing in a place like this (poor data, ordinary
  methods, policy makers who are not interested)?

18
References

• Amati et al (2007) The Importance of Factor
  Levels in the forecasting system for the
  expenditure of public investments Asrdlf 44 th
  Congress, Ersa 47 th Congress (http//www.dps.tes
  oro.it/internship_uver/docs/D420-20Importance20
  of20factors20on20the20expenditure.pdf)
• Bamberger (2006) Conducting Quality Impact
  Evaluations under Budget, Time and Data
  Constraints, Independent Evaluation Group, World
  Bank
• Nonie Impact Evaluation Guidance, World Bank,
  Subgroup 1 and 2
• UE Working document n.6 Measuring Structural
  Funds Employment Effects March 2007

19

• Thank you for your attention
• For contact e-mail to p.severati_at_isfol.it