Telling policy stories: An ethnographic study of the evidence-policy link

1
Telling policy storiesAn ethnographic study of
the evidence-policy link
Alex Stevens University of Kent
2
Two epigraphs on evidence-based policy

• Nobody rational could possibly want a government
  based on any other type of policy-making.
• Colin Blakemore, quoted in The Observer, November
  2009
• There is no (rational or empirical) scientific
  procedure of any kind whatsoever which can
  provide us with a decision...
• Max Weber, The Methodology of the Social Sciences

3
Research question

• How do policy makers use evidence?
• Or rather
• How do some particular policy makers use evidence
  in a specific policy area (illicit drugs and
  crime)?
• Definitions
• Policy making
• The process of choosing ends and means for
  government action, whether rational or not.
• Evidence
• What counts as evidence to be treated as an
  empirical question.

4
Method and ethics

• Method
• Six months of participant observation in a team
  of English policy making civil servants.
• During a placement funded by ESRC.
• Five semi-structured interviews.
• Analysed through adaptive coding of fieldnotes
  and transcripts (Layder 1998).
• Ethics
• Approved by internal ethics committee.
• Negotiated in the field.
• Informed consent from closest colleagues and
  interviewees.
• Covert observation of others.
• Anonymisation of all participants.

5
Prevalent models and approaches to the
evidence-policy link

• Models
• Linear
• A direct link between research evidence and
  government policy.
• Political/tactical
• Evidence is used in support of short-term
  political interests.
• Enlightenment (Weiss 1976)
• Evidence informs the climate of opinion in which
  decisions are taken.
• Approaches
• Pluralist (Kingdon, Sabatier Jenkins-Smith,
  Nutley).
• Post-foundationalist (Hajer, Valentine).
• Critical (Bourdieu, Habermas).

6
The research environment

• A small team of people, most of whom were moving
  quickly between jobs in search of rapid
  promotion.
• Mostly non-specialist, fast tracked civil
  servants, with some secondees.
• Social backgrounds
• Mostly white British.
• Middle class.
• Male.
• Young (lt35).
• Advising the highest levels of UK (English)
  government on crime and drug policy.

7
Seven themes and two cases

• Themes
• The commitment to evidence.
• The oversaturation of evidence.
• Intra-government relations.
• From contested norms to monetary facts.
• The control of uncertainty.
• The preference of intuition.
• Bureaucratic competence and the civil service
  career.
• The third face of bureaucratic reason.

• Cases
• The silent silencing of inequality
• The call for totemic toughness

8
1. The commitment to using evidence

• The civil servants were highly committed to the
  use of evidence
• its the job of officials to tell truth to
  power
• evidence is a prerequisite for policy
• evidence should be the basis for options we put
  to ministers evidence-based policy is part of
  the way that we work.
• All could give examples of policies that had used
  evidence.
• But all could also give examples of where it had
  not.

9
What counts as evidence?

• Fifteen types of evidence coded from
  observations
• internally collected government data
• reports by thinktanks (e.g. ippr, Demos, Policy
  Exchange, Centre for Social Justice, etc.)
• opinion polls
• reports from management consultancies.
• previous policy papers
• independent inquiries
• reports of the inspectorates of police and
  prisons
• internal evaluations
• external evaluations
• reports from abroad
• press reports
• externally produced academic analysis
• television programmes (e.g. The Wire)
• personal or reported experience
• personal or reported opinion
• Mostly found through Google searches
• No on-site library or online journal access.

10
2. The oversaturation of (unsuitable) evidence

• a depressingly similar pattern where you
  look for the best - usually quantitative data -
  you can find, and then, as you work through the
  policy problem you establish that there is not
  the best evidence that you want and you work your
  way down until, at the end, youre left with the
  odd case study, something which was kind of half
  evaluated, some anecdotal information and then
  what you can garner through a few field visits.
• (Policy adviser)

11
3. Intra-government relations (and the creation
of policy stories)

• The point of policy work is to get policies
  accepted.
• Requiring iterative negotiation within government
• Effectiveness becomes a secondary consideration.
• This requires the creation of coherent narratives
  to sell the policy to colleagues and ministers
  across departments.
• Persuasive narratives
• chime with existing ways of thinking.
• exclude uncertainty.
• For example

12
The excision of caveats
13
4. From contested norms to monetary facts

• Repeated attempts to convert contested questions
  of normative value into simple questions of
  financial value
• Fairness cant be measured. Its irrelevant.
  (Civil service economist)
• Derision of old-style civil servants for
  muddying the water with questions on fundamental
  policy aims.
• Counter-example
• Use of humane values to question continuing the
  high rate of youth imprisonment.
• But did not eventually affect the policy.

14
5. The control of uncertainty (through the use of
killer charts)

• A good chart is worth ten pages in words
  (Senior civil servant)
• Killer charts.
• The prevalent form of communication for policy
  related, quantitative evidence.
• Restricted number of variables and cases.
• Preferably dramatic.
• A visual rendition of statistics which renders
  invisible the process of construction of these
  statistics.
• For example...

15
(No Transcript)
16
On current projections, justice is going to
become unaffordable
1. Context for reform
Actual and projected total MoJ spend (bars)
against actual and potential CJS demand (line),
millions
Even if MoJ total spend is flat in real terms,
CJS spend could swallow 90 of MoJ spending by
2019/20, compared to 76 in 2008/9.
This graph assumes that CJS demand continues to
grow at the average rate of the last six years,
and that MoJ total spending follows the IFS
equal pain scenario.
Spend on the criminal justice system includes
NOMS, prisons, probation, YJB, OCJR, criminal
injuries compensation, and the proportions of
legal and HMCS budgets that are spent on criminal
justice
17
6. The preference for intuition over evidence

• A longstanding preference
• My colleagues have an almost unanimous reliance
  on intuition and a distrust of systematic
  argument (civil servant quoted by Garrett, 1972)
• Exemplified by the lack of need for evidence for
  market solutions to public problems
• E.g. Purchaser/provider split in NOMS justified
  by Ministers on the basis of Carter reports
  unevidenced preference.
• it just feels intuitively right that introducing
  competition would focus more on cost and quality
  (Senior civil servant)

18
7. Bureaucratic competence and the civil service
career.

• The three unwritten rules of civil service
  promotion
• 1. Do not specialise.
• Never less than two years, never more than
  three (quote on departmental intranet chat
  page).
• Specialist experience is seen as a positive
  disadvantage (policy adviser, pub conversation)
• 2. Find superior supporters.
• its supposed to be open and equal but its not.
  People go for people they know. If youve worked
  with someone senior, you try and stay in touch
  with them they can help you (female policy
  adviser).
• 3. Be useful.

19
On being useful

• I found a problem with policy area. My boss
  said Well youre young. Why dont you suggest we
  look again at policy area and see how far that
  takes you in your career? So there are certain
  areas where officials will self-censor and they
  wont suggest to ministers to change policy on
  certain areas even though the evidence suggests
  it.
• I think if you always use the evidence when it
  conflicts with current policy then you're always
  going to be the awkward person thats saying,
  the Emperor has no clothes.
• Policy adviser, grade 7.

20
8. The third face of bureaucratic reason

• First face
• Using reason and empirical evidence to create
  useful knowledge.
• E.g. Discussion on imprisonment at Ministry of
  Justice
• Second face
• Using the performance of this rational capacity
  to demonstrate worthiness for status and reward.
• E.g. Competitive performance of knowledge during
  this discussion.
• Third face
• Using only a certain selection of the range of
  rationally justifiable positions in order to
  demonstrate usefulness, to become a protégé and
  to qualify oneself for acceptance and promotion.
• E.g. Omission of explicit recommendations to take
  direct action to reduce imprisonment from
  eventual policy proposals.

21
Case A The (ignored) importance of inequality
22
(No Transcript)
23
(No Transcript)
24
The silent silencing of inequality

• Wilkinsons presentation described as
  compelling and convincing, but no action
  taken on inequality in my policy area.
• Actions that are taken on drugs and crime
  reinforce class and ethnic inequalities.
• the Gini coefficient is not a policy lever that
  we can pull (Senior civil servant, grade 5)
• We need to keep the lid on (same civil servant)

25
Case B totemic toughness

• We need to come up with totemic, tough policies
  
• (Special Adviser and Senior and other Civil
  Servants)
• Symbolic fragmentation/expurgation
• The job of policing is to protect the goodies
  and to catch the baddies (Plenary speech at a
  policing policy conference)
• we know who were talking about. Its not the
  public schoolkids waiting at the bus stop, its
  those other kids. (Civil servant during meeting
  on incivility)

26
Limitations

• Ethnography provides a deep but narrow picture of
  empirical reality.
• Different governments may do things differently.
• Other civil servants may have acted differently.
• Other policy areas may operate by different
  implicit rules on the use of evidence.
• A different researcher may have created different
  reactions and noticed different behaviours.
• The shaping of persuasive research narratives.
• E.g. the dangerous attraction of the killer quote.

27
So how do policy makers use evidence?

• For these policy makers
• Evidence was used in the creation of policy
  stories that favoured action over contradiction,
  and certainty over accuracy.
• Evidence was more likely to be used if
• If it did not challenge existing asymmetries of
  power and wealth.
• If it totemically reinforced the ideological
  fragmentation of society into deserving and
  undeserving citizens.

28
Civil Service Reform Panglossian pluralism?

• No acknowledgement of systemic bias.
• Reiterations of need to base policy on sound
  evidence
• while continuing to roll out unevidenced
  policies insert your favourite here.
• and continuing to miss opportunities to design
  innovations to generate valid knowledge of
  effect.
• the key test of good policy is the feasibility
  of implementation
• Naïve faith in open policy making
• with an added dash of privatisation
  (contestability)

29
Conclusions

• Policy makers are faced with an unreadable deluge
  of unconvincing evidence.
• They use some of this evidence to tell and sell
  persuasive policy stories.
• Which get policy accepted.
• Which help to advance their careers.
• Selection of evidence for these stories fits the
  interests of powerful groups and so represents a
  form of ideology
• Systematically distorted communication
  (Habermas, 2002)

30
Acknowledgements