The Monitoring and Evaluation of Government Policy: Appropriate Alignment Across the Spheres and Fun

1
Monitoring and evaluating policy across the
spheres and functions of government
University of Cape Town Department of Statistical
Sciences
Funding was contributed by the Conflict and
Governance Facility, a project of National
Treasury, which is funded by the European Union
under the European Programme for Reconstruction
and Development
2
ME

• Monitoring A continuing function that uses
  systematic collection of data on specified
  indicators to provide management/stakeholders
  with indications of the extent of progress and
  achievement of objectives
• Evaluation The systematic and objective
  assessment of an ongoing or completed project,
  programme or policy, its design, implementation
  and results. The aim is to determine the
  relevance and fulfilment of objectives,
  efficiency, effectiveness, impact and
  sustainability.
• Adapted from OECD http//www.oecd.org/dataoecd/2
  9/21/2754804.pdf 17 Oct 2005

3
ME

• Motherhood and apple pie? Is there anything to
  discuss?
• Does access to data lead to better information,
  lead to better decision making, lead to better
  performance?

4
Evidence of what?

• Monitoring involves selecting an indicator that
  can provide evidence of progress towards some
  objective (value judgements)
• Assumptions causality and the links between
  actions and results.
• To provide evidence
• Articulate the framework within which monitoring
  will occur.
• Clarify the logic that will be applied in making
  the claim for evidence (of progress).

5
ME of Policy in South Africa Alignment across
the Spheres and Functions of Government

• Purpose
• To develop a critical appraisal of the current
  system of monitoring and evaluation established
  by government around the Programme of Action
  (POA)
• To propose a set of indicators that can be used
  to appraise performance.
• To investigate the alignment of ME processes
  across national and provincial spheres of
  government.-
• To investigate the existence of appropriate data
  sets that can be used to calibrate indicators.

6
Successful ME systems

• Build an understanding of why ME is essential
  and helpful (demand)
• Complex role of incentives
• Leadership is key
• Start by diagnosing existing flows of information
• Only collect what will be used
• Build credible data/ admin systems
• Dont over-engineer the system
• Build in structural objectivity (data integrity)

7
Goal setting and prioritisation

• MCDA

• Problem Structuring defining the problem
  arena, including stakeholders
• Specifying criteria (in ME these could be
  referred to as objectives, goals or outcomes)
• Development of alternative courses of action
• Preference modelling
• Assigning weights to the criteria

8
A strategic framework for ME

• Articulating Goals/ Values/ High level Objectives
  (or, put simply what is the problem?)
• Developing a theory of change on how
  interventions will tackle the problems within the
  problem context
• Determining what activities will be undertaken to
  address the problems.
• Outcomes if all interventions are successful
  where will we be? The answer is the (medium to
  long-term) outcome. NB Check that activities
  outcomes objectives/ goals.

9

• Strategic ME

Problem
Ind
Inputs
Theory of Change
Objectives/ Goals
Ind
Activities Programmes projects, etc
Ind
Ind
Outputs
Does outcome lead to goal being fulfilled?
Outcome
Ind
10
High level objectives for the Province
A Home for ALL
To ensure the integrity of the physical
environment
To enhance economic participation, shared growth
and development
To enhance spatial integration and provide
sustainable service delivery
To develop social and human capital
To practice holistic governance and administratn
11
A set of desired outcomes for each objective
12
A Home for ALL
To ensure the integrity of the physical
environment
To enhance economic participation, shared growth
and development
To enhance spatial integration and provide
sustainable service delivery
To develop social and human capital
To practice holistic governance and administration
Enhanced broad-based economic participation
Enhanced provincial economic growth
black ownership of companies
real GDPR growth per capita p.a.
Pro-poor, shared economic growth
Skills profile which matches the needs of the
economy
Share of poorest quintile in provincial
consumption (MDG).
Adequate spatial integration and transport
infrastructure to support a growing and
integrated economy
of Science, Engineering and Technology graduates
Significantly increased employment
of individual income spent on transport
Unemployment rate
13
A Home for ALL
To ensure the integrity of the physical
environment
To enhance economic participation, shared growth
and development
To enhance spatial integration and provide
sustainable service delivery
To develop social and human capital
To practice holistic governance and administration
Spatially integrated, sustainable human
settlements
(of poorest quintile) living in approved ISHS
Basic services and adequate bulk infrastructure
which leads to improved quality of life
of households (poorest quintile) accessing
safe water source
14
A Home for ALL
To ensure the integrity of the physical
environment
To enhance economic participation, shared growth
and development
To enhance spatial integration and provide
sustainable service delivery
To develop social and human capital
To practice holistic governance and administration
A physical environment which enhances quality of
life
Incidence of water borne disease/ flooding/ shack
fires
An environment which promotes sustainable
livelihoods
(of poorest quintile) sustained through fishing
quotas/ subsistence farming/ ecotourism
Sustainable access to safe energy resources
Proportion of total energy expenditure in
province coming from renewable energy sources
Sustainable access to safe water resources
Fresh (dam) water supplies per capita
15
A Home for ALL
To ensure the integrity of the physical
environment
To enhance economic participation, shared growth
and development
To enhance spatial integration and provide
sustainable service delivery
To develop social and human capital
To practice holistic governance and administration
High levels of social capital in communities
Improved human capital and educational status
Social Capital Index
An improved health profile
Literacy rates
Average life expectancy
Safe communities
Proportion of schools located in identified high
crime areas
An effective social safety net
population dependent on social wage as only
source of income
16
A Home for ALL
To ensure the integrity of the physical
environment
To enhance economic participation, shared growth
and development
To enhance spatial integration and provide
sustainable service delivery
To develop social and human capital
To practice holistic governance and administration
Government operates in a holistic
manner Constitutional rights are
protected Government is effective, efficient and
has good customer (citizen) relations There is
political stability and peace There is democracy
and accountability There is adherence to the rule
of law Government is ethical and free of
corruption
Annual qualitative assessment
17
A Home for ALL
To ensure the integrity of the physical
environment
To enhance economic participation, shared growth
and development
To enhance spatial integration and provide
sustainable service delivery
To develop social and human capital
To practice holistic governance and administration
High levels of social capital in communities
Improved human capital and educational status
Safe communities
Basic services and adequate bulk infrastructure
which leads to improved quality of life
An effective social safety net
An improved health profile
Spatially integrated, sustainable human
settlements
An environment which promotes sustainable
livelihoods
A physical environment which enhances quality of
life
Sustainable access to safe water resources
Sustainable access to safe energy resources
Government operates in a holistic
manner Constitutional rights are
protected Government is effective, efficient and
has good customer (citizen) relations There is
political stability and peace There is democracy
and accountability There is adherence to the rule
of law Government is ethical and free of
corruption
Adequate spatial integration and transport
infrastructure to support a growing and
integrated economy
Enhanced provincial economic growth
Pro-poor, shared economic growth
Enhanced broad-based economic participation
Significantly increased employment
Skills profile which matches the needs of the
economy
18
National value tree
19
A society that truly cares a better life for
all (as in the Bill of Rights Constitution)
Create a single integrated economy that
benefits all.
Build a healthier nation
Help consolidate peace, security development in
southern Africa, Africa globe Position SA as
an effective global player
Improve safety security
Broaden access to services consti-tutional
rights and provide good governance
Promote social cohesion (14) a non-racist,
non-sexist, democratic society
Alleviate poverty
Reduce unemployment
Continue economic growth (6, 14, 22)
Strengthen democracy esp. Zimbabwe, DRC, Angola,
also Congo, Swaziland also Tanzania, Kenya,
Burundi, Rwanda, Ghana, Senegal, Ivory Coast,
Sudan, Ethiopia, Somalia, Mahgreb) (60, 61)
Facilitate economic activity within the second
economy, Pro-mote involvement of marginalised in
economic activity (27-39)
Improve opportunities for self-employment
Promote sustainable community livelihoods (27)
Provide comprehensive social security (10) and
ensure people access them (6)
Facilitate creation of jobs, skills and work
experience (27,29,32-36,43,45)
Pursue multilateral global approach. Democratise
UN, IMF, WB, etc. Improve relations with China
(64)
Reduce Corruption (10)
Improve household community assets
Reduce death from road accidents
Address social roots of criminality
Create new jobs
Build community, good citizenship, solidarity
Improve services in government
Better interaction between govt people (10, 50,
71)
Better inter-govt. cooperation
Better implementation, monitoring, evaluation
Integrate traditional leadership
Better access to govt information
Strengthen democratic institutions incl. courts
Facilitate broad-based BEE
Ensure low interest inflation rates
Speed up rural development urban renewal
Improve efficiency of justice system (9,55)
Reduce crime m. death from crime
Promote growth of small micro-enterprises
(30,31)
Provide the skills re-quired by the economy
(27,29,32-36,43,45)
Invest in road, rail air transport,
telecommunications energy
Increase public private of investment (24-26)
Create a compassionate govt service (9)
Improve state capacity for growth development
(47)
Improve provision of information (49)
Strengthen SAs Early Warning Mechanism
Economic cooperation in southern Africa
Realise NEPAD, strengthen AU, SADC (62)
Cooperation with South
Cooperation with industrialised nations
Efficient logistics and infrastructure (59)
Speed up delivery of water, sanitation, phones,
health, electricity, housing, education
Multipurpose centres, e-govt, etc. (49)
Improve local govt capacity (48)
Provide free basic water electricity
Provide efficient, safe transport (9,26, 52)
Ensure people access grants
Improve UIF
Allocate more resources to welfare grants for 3
yrs (44)
Comprehensive care treatment of HIV AIDS.
Expand primary school nutrition progm (43)
Expand primary health care (42)
All children with decent classrooms
Reduce teacher/pupil ratio
Spend in favour of students from poor households
Expand school nutrition program.
Introduce national health insurance
Free health service to disabled
Protect borders (illegal immigration, crime,
terrorism, marine resources)
Reduce awaiting trial numbers
Strengthen prosecution Scorpions, special courts
Improve coordination among relevant agencies
More, better trained police
Improve anti-corruption structures
Intervene in crime areas-enhance normal communal
life (53)
Provide efficient, safe road rail transport (9,
26, 52)
Build more subsidised housing, medium den-sity
housing closer to work provide serviced stands
20
A society that truly cares a better life for
all (as in the Bill of Rights Constitution)
Create a single integrated economy that
benefits all.
Build a healthier nation
Help consolidate peace, security development in
southern Africa, Africa globe Position SA as
an effective global player
Improve safety security
Broaden access to services constitutional
rights and provide good governance
Promote social cohesion a non-racist,
non-sexist, demo-cratic society
Alleviate poverty
Reduce unemployment
Continue economic growth
21
A society that truly cares a better life for
all (as in the Bill of Rights Constitution)
Create a single integrated economy that
benefits all.
Reduce unemployment
Create new jobs
Improve opportunities for self-employment
Promote sustainable community livelihoods
Promote growth of small micro-enterprises
Facilitate creation of jobs, skills and work
experience
Facilitate economic activity within the second
economy, Pro-mote involvement of marginalised in
economic activity
Comprehensive assistance to emerging farmers,
expand agricultural support programs
Ensure involvement of communities in local
economic initiatives
Encourage labour intensive methods
Deal with negative impact of casualisation
Extend Expanded Public Works Program
Change to financial system and institutions
Education system geared for productive work
Support micro-loan financing
Reform support agencies
Learnerships
Skills training
Complete land restitution
Continue land reform
Provide efficient, safe transport
22
A society that truly cares a better life for
all (as in the Bill of Rights Constitution)
Build a healthier nation
Reduce preventable causes of death-TB, diabetes,
malnutrition, maternal deaths, HIV/AIDS, malaria
Recruit retain health personnel
Improve services in health facilities
Complete tertiary infra-structure improvements
Comprehensive care treatment of HIV AIDS.
Expand primary school nutrition program
Expand primary health care
23
A society that truly cares a better life for
all (as in the Bill of Rights Constitution)
Broaden access to services constitutional
rights and provide good governance
Improve services in govt
Better interaction between govt people
Better inter-govt. cooperation
Better implemen-tation, monitoring, evaluation
Integrate traditional leadership
Better access to govt information
Strengthen democratic institutions incl. courts
Create a compassio-nate govt service
Improve state capacity for growth develop-ment
Improve provision of information
Reduce corruption
Multipurpose centres, e-govt, etc.
Improve local govt capacity
24
A society that truly cares a better life for
all (as in the Bill of Rights Constitution)
Create a single integrated economy that
benefits all.
Alleviate poverty
Reduce unemployment
Improve household community assets
Improve opportunities for self-employment
Promote sustainable community livelihoods
Speed up delivery of water, sanitation, phones,
health, electricity, housing, education
Comprehensive assistance to emerging farmers,
expand agricultural support programs
Provide free basic water electricity
Complete land restitution
Continue land reform
Links to GW ME system
Build more subsidised housing, medium density
housing closer to work provide serviced stands
The following is the heading and definition for
ALL of the indicators mentioned on this
page Housing These indicators measure the
marginality of human living conditions
(e) Proportion of HH with access to secure tenure
(owned or rented) (Target 10 Indicator 32)
(a) Percentage of population housed in
informal/formal dwellings. (b) Percentage of the
population which is homeless.
(a) Proportion of population with access to an
improved water source (both urban and rural)
(Target 10 Indicator 30) (b) Proportion of
population with access to improved sanitation
(Target 10 Indicator 31) (c) of gdp spent on
sanitation infrastructure (d) Proportion of
households with access to electricity
25
A society that truly cares a better life for
all (as enshrined in the Bill of Rights,
constitution)
Create a single integrated economy that
benefits all.
Build a healthier nation
Alleviate poverty
Reduce unemployment
Continue economic growth (6, 14, 22)
Facilitate economic activity within the second
economy, Pro-mote involvement of marginalised in
economic activity (27-39)
Improve opportunities for self-employment
Promote sustainable community livelihoods (27)
Provide comprehensive social security (10) and
ensure people access them (6)
Facilitate creation of jobs, skills and work
experience (27,29,32-36,43,45)
Improve household community assets
Create new jobs
Facilitate broad-based BEE
Ensure low interest inflation rates
Speed up rural development urban renewal
Promote growth of small micro-enterprises
(30,31)
Provide the skills re-quired by the economy
(27,29,32-36,43,45)
Invest in road, rail air transport,
telecommunications energy
Increase public private of investment (24-26)
Speed up delivery of water, sanitation, phones,
health, electricity, housing, education
Provide free basic water electricity
Provide efficient, safe transport (9,26, 52)
Comprehensive assistance to emerging farmers,
expand agricultural support programs (37-39)
Ensure involvement of communities in local
economic initiatives
Learnerships
Education system geared for productive work
Skills training
Complete land restitution
Continue land reform
Ensure people access grants
Improve UIF
Allocate more resources to welfare grants for 3
yrs (44)
Encourage labour intensive methods
Deal with negative impact of casualisation
Extend Expanded Public Works Program (28-29)
Support micro-loan financing
Reform support agencies
Change to financial system and institutions
Comprehensive care treatment of HIV AIDS.
Expand primary school nutrition progm (43)
Expand primary health care (42)
Provide efficient, safe transport (9,26, 52)
All children with decent classrooms
Reduce teacher/pupil ratio
Spend in favour of students from poor households
Expand school nutrition program.
Introduce national health insurance
Free health service to disabled
Build more subsidised housing, medium den-sity
housing closer to work provide serviced stands
26
Measurement issues

• Simple vs composite indicators
• Parsimonious models
• Robust or sensitive to change
• Local or global views
• Error (measurement, sampling, statistical bias,
  interpretation)
• Measuring (step 1) and valuing (step 2)

27
Some conclusions

• Pervasive lack of clarity on high level
  objectives value trees might be a useful tool
• Priority. Whats that?
• Danger of over-engineering
• Danger of separating ME from strategic planning
  (activities may not be aligned with desired
  outcomes)
• Tremendous benefits of awareness created by
  MTEF. but
• Lack of quantitative/ problem solving/ modeling
  skills

28
Strategic ME

• There is a strong need for a common strategic
  planning process which is individually tailored
  by each department/ govt entity within the
  context of clear national, provincial, local
  objectives.

29
work that has arisen from this project

• Can we cluster municipalities as red, amber,
  green?
• Determining a snapshot of municipal performance
• Developing an ME framework for the province as a
  whole and individual departments
• Education department of WC establishing an
  integrated and strategic approach to implementing
  ME wrt the literacy and numeracy strategy