Elements of poverty reduction program monitoring and evaluation

1
Elements of poverty reduction program monitoring
and evaluation
Sophisticated development context the Kazakh
experience
Ayse KudatSocial Assessment2004
2
WHAT IS OUR CHALLENGE IN KAZAKHSTAN?

• START A PRPM PROCESS
• DO SO BY TAKING MODEST SUSTAINABLE STEPS
• AS A FIRST STEP, INTEGRATE NATIONAL AND OBLAST
  LEVELS PRP ELEMENTS INTO A UNIFIED SYSTEM AND
  SUPPORT THIS INTEGRATION WITH REAL-TIME AND
  GEPOGRAPHICALLY SPECIFIC COMPUTER APPLICATION
• DESIGN A PROCESS OF CAPACITY BUILDING FOR PRP AND
  PRPM BY LINKING INPUTS, ACTIVITIES, OUTPUTS,
  OUTCOMES AND IMPACTS AT THE OBLAST LEVEL
• INITIATE A SIMPLE SYSTEM OF PRPM COORDINATION AND
  CREATE PRP MONITORING WORKING GROUPS IN MEBP,
  MLSP AND IN THE 14 OBLASTS
• ENSURE MONITORING CONTRIBUTIONS OF NGOS CURRENTLY
  COOPERATING WITH OBLASTS IN PRP PREPARATION AND
  IMPLEMENTATION
• SUPPORT THIS PROCESS IN 2 PILOT OBLASTS WITH
  FOCAL POINTS FINANCED BY THE TA PROJECT
• SUPPORT THE PARTICIPATORY PRPM GOALS OF OBLASTS
  BY HELPING 2 PILOT REGIONS STRENGTHEN ME
  CAPACITY IN THE RAYONS---HELP THESE OBLASTS
  ENHANCE LOCAL ME CAPACITY WITH MODEST RESOURCES
  AVAILABLE IN TA PROJECT
• CREATE INSTITUTIONAL ARRANGEMENTS FOR EVALUATION
• CONDUCT PILOT STUDIES FOR IMPACT ASSESSMENT AND
  INITIATIVE SURVEYS TO TRACK PUBLIC EXPENDITURES
• ASSESS EQUITY AND EFFICIENCY IMPLICATIONS OF
  PUBLIC SPENDING ON PRP
• EVALUATE IMPLICATIONS OF TA PROJECT ACTIVITIES
  FOR NATIONWIDE IMPROVEMENTS IN THE ME OF PRPS
  AND PRP

3
Q WHAT DO WE LEARN FROM INTERNATIONAL EXPERIENCE
OF POVERTY REDUCTION STRATEGIES/PROGRAMS?

• A PUBLIC SECTOR EXPENDITURES AND ACTIVITIES
  DEVOTED TO POVERTY REDUCTION PROGRAMS (PRP)
  CANNOT BE JUSTIFIED OR CAUSALLY LINKED TO POVERTY
  TARGETS UNLESS A RESULTS BASED MONITORING IS
  ESTABLISHED

4
In Kazakhstan we know more about outcomes than
about the means of achieving them
Results Monitoring
Implementation Monitoring (Means and Strategies)
5
Sources of Information for PRPM
TA financed qualitative surveys, Public
expenditure Tracking Surveys in 2 pilot oblasts,
Households surveys, qualitative surveys Public
expenditure Tracking Surveys, Sector specific
surveys
community infrastructure inventories, Rayon MO
achievements, WSS Systems repaired/constructed
Oblast PRP workshops, task forces, Policy
reviews,
Oblast budgets, oblast payroll, Rayon budgets,
rayon payroll
6
Progress made in PRPME in Kazakhstan

• 1. Conducting readiness assessment
• 2. Agreeing on Outcomes to monitor
• 3. Selecting indicators to monitor outcomes
• 4. Establishing baseline
• 5. Selecting realistic targets
• 6. managing for results
• 7. Role of evaluations
• 8. Reporting findings
• 9. Using findings
• 10. Sustaining ME systems

7
Much is already known

• What data are collected (source)
• When data are collected (frequency)
• How data are collected (methodology)
• Who collects the data
• For whom the data are collected
• Who reports the data

Where revenues Are collected, Expenditures
made And activities held, Outcome data Are not
systematically gathered
RAYONS
8
Rayon Administrations
Local Univ., and civil society organizations
Oblast PRP working groups, with Oblast ME
Coordinators
Households
CITIZEN FEEDBACK
Two Pilot Oblasts
Remaining Oblasts
Focal Points
Ministry of Labor and Social Protection
MEBP Inter-Ministry Steering Committee
Secretariat Monitoring Working Group
Joint evaluation Group
Other Line Ministries
National Statistics Agency
Steering Committee
Recommendations to the Parliament and the Cabinet
for Poverty Reduction Strategy
9
Stakeholders in KAZAK PRS monitoring

• Central Ministries (Finance, Economy)
• MEBP consolidate information,
• Line Ministries (MLSP)
• Provide oblast based data, helps evaluate results
• Reporting arrangement to be structured between
  MEBP and other line Ministries under the TA
• Universities, research institutes
• Local universities contribute to training and
  impact data gathering for small scale impact
  studies
• Rayon governments
• Key provider of input and output data to Oblasts
• National statistical agencies
• Conducts reliable and large scale, nationwide
  household surveys representative at the oblast
  level
• Develops rayon level capacity for collection of
  output data
• Civil society organizations
• Trade unions provide feedback information,
  Association of SME can contribute impact data
• Parliament/Cabinet
• MEBP through the Steering Committee to carry out
  Evaluation and provide strategy feedback to
  Parliament and Cabinet

10
Participation and Feedback

• Civil society involvement in policy formulation
  exists at Oblasts and through the TA financed
  focal points their contributions for ME can be
  structured.
• Through PRPM the program planning for future PRPs
  (2004) can be improved. Targets, inputs, outputs
  can be better defined.
• Sharing PRP results with public through the
  Newsletter of the national NGO consortium, as
  well as the website of the MEBP (with links to
  other line Ministries) can help generate citizen
  feedback and ensure sharing of PRP/PRS information

11
Client Feedback

• Limited resources available for all studies,
  including report cards, are extremely limited
  under the TA.
• Specific citizen feedback will be generated
  through the NGO Newsletter
• Citizen/user feedback will be assessed for 2
  projects (e.g., a- SME development for the
  unemployed, and b- micro credit for rural areas)
  in 2 pilot oblasts through surveys carried out by
  local universities. These studies will also
  include budget tracking surveys
• Qualitative impact studies of pioneering programs
  in 2 pilot oblasts (a-process of social passport
  issuance and targeting and b- emigration from
  ecological disaster areas)
• PRPM works shops in 2 pilot oblasts with rayon
  and select settlement representatives
  participatory evaluation of PRP as an input to
  next steps of PRP preparation.

12
Types Of Evaluation Proposed within TA project
13
Using the Results - Information Dissemination

• Results will be disseminated through MEBP website
• Results will be disseminated through MLSP website
• Targets, inputs, outputs, outcomes will be
  disseminated through newsletters published by
  civil society groups
• MEBP will hold regular press conferences and
  issue press releases on PRP progress
• Joint evaluation groups will be established at
  the central and pilot oblast levels
• Inter-ministry information will be assured
  through the Steering Committee as well as through
  ME data flow yet to be structured from line
  ministries to the MEBP

14
Who Monitors? How should Participation in PRPM
be organized?

• Many PRPs rely on outcome monitoring
• Such a system largely available in Kazakhstan
• But does not allow evaluation of the PRS and PRP
• Runs the risk of being a paper exercise
• Responsibility for PRP Monitoring is with the
  MEBP
• MLSP shares the burden
• Other Ministries can contribute
• Agency for Statistics tracks quarterly trends in
  poverty outcomes
• Oblasts, rayons contribute to ME
• Civil society organizations support ME and can
  do so more systematically and regularly
• Citizen involvement in PRPM can be improved
• Public Expenditure Tracking Systems (PETS)
• Quantitative Service Delivery Systems (QSDS)
• Social Assessments, Citizen Report Cards
• Participatory Impact Monitoring

15
Monitoring data flow

• In terms of institutional mechanisms, the data on
  poverty will flow through several different
  channels
• From RAYONS to OBLASTS to the central level MEBP
  for monitoring PRP
• From RAYONS to Statistical Agency for monitoring
  of the outcomes Sectoral information, MIS data
  and/or data from sample surveys conducted by
  different central agencies, including the MLSP to
  MEBP
• Data from key NGOs (e.g., trade unions,
  association of business and farmers association)
  to Oblast governments and transmitted to MEBP
• Case studies, budget tracking studies and client
  satisfaction studies conducted at the Oblast
  level and shared with MEBP

16
Real-time Monitoring and Evaluation Software
Sources of Monitoring Indicators

• Input, Activity and Output Indicators Unified
  and re-prioritized in accordance with the
  National Poverty Reduction Strategy
• Data obtained from the oblast level
• Data obtained from the implementing line
  ministries
• Outcome Indicators
• Data obtained from the oblasts
• Data obtained from the National Statistics Agency

Expected Results from the Monitoring Software

• Analysis of specific actions underlined at the
  National Poverty Reduction Strategy in terms of
  actual outputs provided, and poverty reduction
  outcomes achieved
• Making these results available at the oblast
  level
• Enabling direct linkages between resources put
  through specific actions and policy outcomes of
  these actions, assisting policy makers in
  designating new priorities

17
Sample Indicators How the Software Will Work
Oblasts
Inputs Name of Activity Primary school
repairs Amount of financing (to be indicated) If
financing not available, indicate estimated
number of man-hours dedicated to this activity
Activities and Outputs Number of schools
undergoing repairs Estimated number of
beneficiaries (ie the number of pupils in these
schools)
National Level
Outcomes Number of primary school age
children studying in schools urban and
rural Percent of pupils in schools that study
on the third shift
Charts and Graphs for summary information and
evaluation
18
Results will be made available on a geographical
system (sample indicators on education)
30 schools repaired 15,000 pupils benefited
20 schools repaired 13,000 pupils benefited
10 schools repaired 9,000 pupils benefited