Slayt 1

1
Bölgesel Rekabet Edebilirlik Operasyonel
Programinin Uygulanmasi için Kurumsal
Kapasitenin Olusturulmasina Yönelik Teknik
Yardim Technical Assistance on Institutional
Building for the Implementation of RCOP in Turkey

This project is co-financed by the European Union
and the Republic of Turkey
Evaluation management Ankara, 15 December
2011 Laura Trofin (PhD) TAT Non-Key Expert ,
Monitoring and Evaluation lauratrofin_at_yahoo.com
2
Presentation Structure
This project is co-financed by the European
Union and the Republic of Turkey

• Planning an evaluation
• Costs and evaluation budgets
• Managing stakeholders
• Role and functions of steering committees
• Terms of reference
• Assessing evaluation tenders
• Managing the evaluator
• Quality assessment

3
Planning an evaluation Timing Issues
This project is co-financed by the European
Union and the Republic of Turkey

• Evaluations take time
• 1 year for some evaluations
• from decision to final report
• Work back from final report deadline
• Allow time for procurement process
• Think about when monitoring data will be
  available
• Allow 4 6 months for draft report
• Risk management
• Things will go wrong
• Get stakeholders and steering committee to agree
  timetable

4
Planning an evaluation Timing Issues
This project is co-financed by the European
Union and the Republic of Turkey
5
Evaluation Plan
This project is co-financed by the European
Union and the Republic of Turkey

• Multi-annual
• Annual
• Types, timing, budget of evaluation
• Feeds into procurement plan, especially for your
  TA
• Makes useful results available when you need it

6
Planning an evaluation Budgets
This project is co-financed by the European
Union and the Republic of Turkey

• Evaluations cost money!
• How much?
• EU Commission recommended 0.5 of programme
  budget
• Evaluation Guide
• 1 for routine programmes
• Up to 10 for innovative, pilot programmes
• Some examples
• NDP/CSF IE 400,000 (292 days)
• OP IE (average of 236,000, 197 days)
• Priority level 163,000 (121 days)
• Policy level 2,000,000 (1200 days)
• CBC IE 60,000 (80 days), 30,000 (35 mandays)

7
Evaluation Planning Stakeholder Identification
This project is co-financed by the European
Union and the Republic of Turkey

• Identify and consult with stakeholders
• The individuals, groups, organisations who have
  an interest in programme to be evaluated
• European Commission
• Finance Ministry
• Managing Authority, Line Ministry
• Implementing Agency
• Interest Groups, NGOs
• Secure their involvement in and cooperation with
  project
• Establish their views on evaluation questions

8
Steering Committee
This project is co-financed by the European
Union and the Republic of Turkey

• What does it do?
• Oversees evaluation project from start to finish
• Key tasks
• Approving terms of reference
• Selecting the evaluator
• Quality control
• Who should be on it?
• Not a consultation forum
• Not all stakeholders, keep it manageable
• Consider involvement of evaluation expert

9
Terms of reference
This project is co-financed by the European
Union and the Republic of Turkey

• A critical part of evaluation process
• Poor TOR lead to poor evaluations!
• TOR should
• Explain the evaluation context
• Ask clear questions
• Identify the key stakeholders
• Indicate what information is available and when
• Say how evaluator will be selected and how
  project will be managed
• Steering committee should approve

10
TOR includes
This project is co-financed by the European
Union and the Republic of Turkey

• Request for tender document includes
• Introduction (summary programme information)
• Evaluation context
• Assumptions and risks
• Purpose, objectives, questions, methods,
  stakeholders
• Description of information sources
• Timing and organisational issues
• Tender requirements (socio-economic, similar
  experience, experts)
• Selection process and award criteria
• Contract conditions

11
TOR Structure
This project is co-financed by the European
Union and the Republic of Turkey
1. Set the evaluation context
2. State the evaluation objective
3. Specify detailed questions
5. Issues for recommendations
4. Issues for evaluation conclusions
12
TOR example
This project is co-financed by the European
Union and the Republic of Turkey

• Step 1 Set the evaluation context
• middle of the implementation process
• Step 2 State clear evaluation purpose
• identify solutions to bottlenecks in the
  programme implementation
• State 3 clear evaluation questions (as presented
  earlier)
• Why has the programme experienced lower demand
  for funds under Priority/Measure x?

13
TOR example
This project is co-financed by the European
Union and the Republic of Turkey

• Step 4 Break evaluation question into analytical
  tasks or sub-questions essential for you,
  evaluator further development
• Analysis of previous evaluation results
• Ex-ante
• Ex-post
• Review of external developments
• Review of policy developments
• Progress to date
• Management and delivery systems
• Horizontal issues

14
TOR example
This project is co-financed by the European
Union and the Republic of Turkey

• Step 5 Summarise the issues on which conclusions
  and recommendations are required REINFORCES,
  not ADDS
• Example
• Contribution of CSF to date and likely impact
• Validity of strategy
• Appropriateness of financial resources
  distribution
• Efficiency of management structures

15
Assessment of Evaluation Tenders Selection
Criteria
This project is co-financed by the European
Union and the Republic of Turkey

• 4 main criteria
• What? analysis and understanding of TOR
  (context, purpose, objectives, questions/sub-quest
  ions)
• How? methodology, timing
• Who? experience and resources allocated
• How much? cost
• Each criterion has same/different weight
• How quantify criteria?
• Interviews sometimes used I recommend

16
Managing the Evaluator
This project is co-financed by the European
Union and the Republic of Turkey

• Keep in touch with the evaluator
• Request regular updates
• Set project milestones
• Inception report
• Steering committee meetings
• Evaluator may need help in
• Establishing contacts
• Getting data

17
Quality Assessment
This project is co-financed by the European
Union and the Republic of Turkey

• How does steering committee assess the quality of
  the evaluation report?
• Evalsed 8 criteria
• Other but evalsed the basis
• Use of independent evaluation experts

18
Quality Control Output criteriaAssessment
This project is co-financed by the European
Union and the Republic of Turkey

• Meeting needs as laid out in ToR
• Relevant scope and coverage
• Defensible design and methods
• Reliable data used
• Sound analysis
• Credible results that relate to analysis and data
• Impartial conclusions showing no bias and
  demonstrating sound judgment
• Clear report with executive summaries and annexed
  supportive data

19
Quality Assurance Process criteria
This project is co-financed by the European
Union and the Republic of Turkey

• Coherent and evaluable objectives 
• Well drawn terms of reference
• Sound tender selection process
• Effective dialogue and feedback throughout
  evaluation process
• Adequate information resources available
• Good management and co-ordination by evaluation
  team
• Effective dissemination of reports/outputs to
  Steering Committee and policy/programme managers
• Effective dissemination to stakeholders

20
Group exercise 5
This project is co-financed by the European
Union and the Republic of Turkey

• Completing the evaluation ToR adding to type,
  criteria, methods, questions the following
  elements purpose, human, time and financial
  resources, planning and expected results.
• THINK ABOUT
• Why you need the evaluation
• When do you need the evaluation results
• How much money you have
• Tools to manage/monitor the evaluation process,
  together with the evaluator and Steering
  Committee
• Their timing/your time

21
This project is co-financed by the European
Union and the Republic of Turkey