Title: This slide presentation includes some slides, marked with , that carry voice recordings. When the slide show is run, the recorded narrative will automatically run.
1This slide presentation includes some slides,
marked with , that carry voice recordings.
When the slide show is run, the recorded
narrative will automatically run. Accompanying
this slide presentation are electronic versions
of all the documents listed in the references.
Moreover, hard copies of some documents,
including this slideshow, are made available to
participants.
2The use and abuse of indicators
- Martin Gustafsson
- October 2012
3Contents
- Introduction
- The history of indicator-based planning
- Concrete examples from the public and private
sectors - The polemics surrounding when and how to use
indicators - A case study Our teacher development indicators
- Some pointers for best practices in the South
African public sector
4Introduction
- We wont spend much time defining what an
indicator is. The working definition for now is
Time series of numbers, generally annual, that
helps planning and reporting. - We also wont look at classifications of
indicators. This is something you have probably
been exposed to. But this session will make
reference to different types of indicators,
including sectoral and organisational indicators.
5Introduction
- This session will be about the selection,
calculation, documenting, presentation and
general use of indicators. - As far as possible, Ive made use of the
available literature. But the literature on the
use of indicators is a bit limited. Above all, it
tends to be theoretical and largely
non-empirical. With regard to the latter it is of
course very difficult to draw any causal links
between planning techniques and a better society.
However, one can examine how planners and the
public actually use and perceive indicators.
Unfortunately, even this kind of analytical work
is difficult to find.
6Introduction
- The session in partly about being honest about
indicators, in particular about whether the
investment we make in this area really makes a
difference to the effectiveness of our
organisations and service delivery on the ground.
So feel free to air your frustrations!
Group discussion What are some of your
frustrations around the use of indicators in
planning and reporting? But also, can you think
of examples where you believe indicators add
value to our work?
7The history of indicator-based planning
- Statistics on whole societies to serve the
interests of rulers and citizens have existed for
centuries. - Governments have played a key role. The word
statistics in fact derives from the word
state. - Ancient civilizations in Asia, Africa and Europe
focussed above all on developing information
systems that would tell government how much tax
to expect from the empire. This included running
population censuses the earliest censuses were
probably those of Ancient Egypt.
8...The history of indicator-based planning
- The modern tradition of national statistical
offices can be traced back to the 1833
establishment of the French national office. This
in turn owes much to the influence of Napoleon,
whose influence in the design of the modern state
is immense. - On the private sector side, the analyst Frederick
Taylor the industrialist Henry Ford, both
Americans, popularised Taylorism and Fordism
in industrial production processes. This involved
measuring the productivity of individual workers
and groups of workers and firing and adjusting
pay accordingly. These approaches were embraced
across the world, even in the Soviet Union.
9...The history of indicator-based planning
- The Great Depression and the Second World War and
their aftermath made governments larger and more
influential. Collection of economic and social
data within countries increased enormously. - On the economic side, work by Keynes influenced
the structure of national accounts covering
aggregate national income and related issues. - In South Africa, Office of the Census,
established around 1910, was the countrys first
national statistics office. Under apartheid,
bantustan offshoots were created, which were
merged into the national post-1994 institution
that eventually became Stats SA.
10...The history of indicator-based planning
- In 1968, the internationally comparable System of
National Accounts (SNA) was started. - New public management (NPM) ideas began changing
rich country governments in the 1980s. The focus
was on bringing private sector type incentives,
performance information and flexibility into
public services. Whether NPM has been eclipsed by
other philosophies such e-Governance is a matter
for debate, but the influence of NPM has
undoubtedly been great.
11...The history of indicator-based planning
- In developing countries, the drive for NPM was
influenced by the logical framework approach
(LFA), or the use of matrices with goals,
outcomes activities and indicators (in other
words what is very common in our government
plans!). LFA has to a large degree been driven by
aid donor agencies.
12...The history of indicator-based planning
- The 1990s saw reporting on social trends
strengthened through the introduction of
international indicators such as the UNDPs Human
Development Indicator (HDI) and a variety of
education indicators by the UIS.
13Concrete examples from the public and private
sectors
- Group discussion
- Look at how social development indicators have
been presented in the three documents below,
which are deliberately from organisations you are
likely not to be familiar with. Think of how
indicators have been selected, how numbers have
been presented and links to the strategy of the
organisation. Think of when one is dealing with
indicator values and when one is dealing with
some other type of statistic.
14...Concrete examples from the public and private
sectors
- ...Group discussion
- The Mauritius Education and human resources
strategy plan 2008-2020. Focus especially on pp.
63-66. - The 2012-2017 strategic plan of the education
department of the Australian state of New South
Wales. Focus especially on p. 10. - The Lonmin Sustainable development report, 2011.
Focus especially on rows under heading
Transformation on pp. 13-14.
15The polemics surrounding when and how to use
indicators
- The attempt here is to focus on those debates
that are most relevant to the South African
public service. - Complaint 1 Indicator frameworks impose a
structure that is too limiting and that prevents
more creative thinking (including creative
quantitative thinking). They become a distraction
rather than a facilitating tool. - Rational, linear planning techniques alone have
proven incapable of successfully introducing and
sustaining effective and efficient reform (Inbar,
1996 16). - A part of this complaint is that impractical
indicators which are in fashion are imposed on
planners - ...there is a tendency to introduce new vogue
indicators, and such measurement manoeuvres place
unnecessary pressure on existing infrastructure,
further disengage staff, and deflect focus away
from well-established, long-term indicators of
performance. (Hailey and Sorgenfrei, 2004 15)
16...The polemics surrounding when and how to use
indicators
- The worst case scenario is that indicators
actually pervert behaviour. This is a very
commonly made complaint. - The Soviet Union, without the benefit of market
prices, even for goods that in capitalist
societies would be described as private goods,
has for decades had to rely on production targets
and performance indicators to control managers.
The potentially disastrous consequences of
specifying an incomplete or ambiguous target are
exemplified by the infamous 'nail' cartoon,
recounted by Mullen ..., in which the factory
fulfils its monthly nail quota with a single
enormous nail, because the success indicator is
specified by weight alone. (Smith, 1990 67) - Analysis that supplements the indicators
themselves is necessary - A system of indicators should work like a control
panel. It facilitates the identification of
problems, and allows for their magnitude to be
measured. Detailed diagnosis and the search for
solutions are done by complementary analysis and
research. (Sauvageot, 1997 17) - Possible response to complaint 1 Dont let
indicators dominate the planning and reporting
processes. Perhaps limit the number of
indicators. Indicators are just one of many
planning elements.
17...The polemics surrounding when and how to use
indicators
- Complaint 2 Indicator frameworks create the
illusion that social sectors work in mechanical
and easily measureable ways. In developing
countries in particular, where data quality is
often poor, it is often impossible to make
reliable year-on-year comparisons. - Comprehensive knowledge, organized into a
coherent framework, gives the appearance of
better control, or reduced uncertainty, and a
decrease of risks. The development of
over-sophisticated, abstract models and plans of
innovation arises from this illusion. Modelling
is often undertaken to persuade politicians or
administrators who demand certainty before
allowing planned change and who too often search
for perfection rather than improvement. ...
models are imbued with almost magical quality
of truth and objectivity. Consequently, there is
pressure to advocate rationality, which may
contribute little or nothing to the change
process, rather than into the development of
human contacts, involvement and commitment which
are true underpinnings of innovation. (Inbar,
1996 91)
18...The polemics surrounding when and how to use
indicators
- Possible response to complaint 2 Indicators
create an incentive to improve the quality of
data. It is true that whilst the data quality is
still poor, indicators are often aspirational
rather than truly implementable and any values
must be interpreted with care.
19...The polemics surrounding when and how to use
indicators
- Complaint 3 If the political tradition is to set
overly ambitious targets, attempts to forecast
the future with very rational indicators
inevitably lead to difficult tensions between
politicians and implementers and, ultimately,
manipulation or marginalisation of actual
indicator values. -
- In the short run, ritualistic plans and
innovations have clear social and political
benefits. Ritualism may have a tranquillizing
effect. (Inbar, 1996 92) - Critically, goals and targets are often set by
those at some distance from those charged with
getting targets. Target setters live in different
worlds to target getters. Ownership at different
levels can be unclear. Delegating responsibility
to achieve targets without delegating authority
and providing control over necessary resources is
often a recipe for frustration and goal
displacement. (Lewin, 2011 5)
20...The polemics surrounding when and how to use
indicators
- It has been argued that implementers may try to
avoid exceptional periods of success just to
avoid having higher targets imposed on them. - Difficulties may also arise when performance
indicators are used to determine managerial
targets .... It is almost inevitable that
principals will set their agents targets on the
basis of previous levels of performance. That
being the case, unless a fairly subtle system of
incentives is introduced, managers have a
continuous incentive to report modest levels of
performance. For to report out-standing
performance in one year would raise the
principal's expectations about future
performance, and therefore extend managers'
targets. This phenomenon, known as the ratchet
effect, has been endemic to Soviet planning ...,
where targets have traditionally been set 'from
the achieved level'. (Smith, 1990 68) - Possible response to complaint 3 Use indicators
to educate everyone, including the politicians,
about what is possible. Perhaps structure reports
in such a way that glaring gaps between actuals
and targets are not too obvious, without lying
about anything.
21...The polemics surrounding when and how to use
indicators
- Complaint 4 The aeroplane cockpit approach to
indicators says that you build the aircraft
first, then fly it whilst reading the dials and
indicators. The problem is that it takes years to
improve measurement practices and data quality.
One is building the plane as one flies it. - Possible response to complaint 4 Yes, one should
accept that initially there will measurement
instability and comparability problems. The
important thing is to admit that initially things
will not be perfect. But one should work towards
measurement that is stable and consistent.
22...The polemics surrounding when and how to use
indicators
- Complaint 5 In reality, no-one ever worries
about indicator values so in the end planners
just put any values into the cells to comply the
rules. - Possible response to complaint 5 Indicators may
become disparaged if they are poorly calculated
and presented. Indicators that speak to the
concerns of managers, politicians and the public
and that contain believable information will be
appreciated.
23A case study Our teacher development indicators
- Group discussion
- Look at how teacher development has been dealt
with in our own national plan and in one
provincial plan. To what extent would the five
complaints we have dealt with apply here? How
could indicators have been employed better in the
two documents? Look not just at the indicators
themselves, but also how they link to the rest of
the plan.
24...A case study Our teacher development
indicators
- ...Group discussion
- The Department of Basic Educations 2012-13
annual performance plan. Look especially at the
teacher development indicators on pp. 36-38. - The Mpumalanga Department of Education 2011-12
annual performance plan. Look especially at p.
37.
25Some pointers for best practices in the South
African public sector
- On the following slide some tentative pointers
are provided to deal with some of the problems
planners who have covered the basics when it
comes to indicators, still experience.
26Some pointers for best practices in the South
African public sector
Calculation of past values
Selection
- The indicator is just the centrepiece related
data analysis must also occur.
- Two points of departure Strategy and available
data.
- Try have a mix of innovation and maintenance
indicators.
- Multiple datasets and slippery values are a
reality one cant just ignore.
- Have few indicators (you can put more numbers
into the narrative).
Narrative
Target setting
- What do comparisons across time and space say?
- Dont torture yourself, its politics!
- What do upward and downward movements mean?
Documentation
- Keep technical notes, you or someone else will
need them later!
- Describe differences between actuals and targets
creatively, but honestly.
- Accept that indicator specifications stabilise
over time.
General use
- Get feedback on the reports, if no-one ever cares
about your indicators theres something wrong.
27References
- All the following sources are in your electronic
source pack - Department of Basic Education (2012). Annual
performance plan 2012-2013. Pretoria. Available
from lthttp//www.education.gov.zagt Accessed
October 2012. - Hailey, J. Sorgenfrei, M. (2004). Measuring
success Issues in performance management.
Oxford INTRAC. Available from
lthttp//www.intrac.org/data/files/resources/53/OPS
-44-Measuring-Success.pdfgt Accessed October
2012. - Inbar, D.E. (1996). Planning for innovation in
education. Paris IIEP. Available from
lthttp//www.unesco.orggt Accessed December 2006. - Lewin, K.M. (2011). Taking targets to task
revisited How indicators of progress on access
to education can mislead. Falmer University of
Sussex. Available from lthttp//www.create-rpc.or
g/pdf_documents/PTA54.pdfgt Accessed July 2011. - Lonmin (2011). Web-based sustainability
development report. London. Available from
lthttps//www.lonmin.com/downloads/pdf/Sustainable_
Development/Lonmin_WBR11.pdfgt Accessed October
2012. - Mauritius Ministry of Education, Culture Human
Resources (2008). Education human resources
strategy plan 2008-2020. Available from
lthttp//planipolis.iiep.unesco.orggt Accessed
December 2009. - Mpumalanga Department of Education (2011).
Annual performance plan for 2011/12. Mbombela. - New South Wales Education Communities (2011). 5
year strategic plan. Sydney. lthttps//www.det.nsw.
edu.au/media/downloads/about-us/how-we-operate/str
ategies-and-plans/corporate-plans/fiveyrs-strategi
c-plan.pdfgt Accessed October 2012 - Sauvageot, C. (1997). Indicators for educational
planning A practical guide. Paris IIEP.
Available from lthttp//www.unesco.orggt Accessed
December 2006. - Smith, P. (1990). The use of performance
indicators in the public sector. Journal of the
Royal Statistical Society, 153(1) 53-72.