Do the Claims for Spending Billions on Crime Reduction Initiatives Stand up

1
Do the Claims for Spending Billions on Crime
Reduction Initiatives Stand up

• Radical Statistics Conference 2006
• Paul Marchant
• p.marchant_at_leedsmet.ac.uk

2
Aims

• A look at knowing What Works.
• Things worth encouraging
• Good Statistics
• (as many arguments are statistical in nature)
• Transparency in design and reporting
• Investigative Statistics
• Scientific scepticism
• Research which is sufficiently sound in order to
  properly justify spending on major programmes

3
Science

• Acquiring knowledge using data from observation
  and experiment.
• An inherently uncertain matterStatistics!
• Not all results given, are the same. Therefore
  there is the need to synthesise.
• Science is a public matter not just because of
  the impact of the products of science, but also
  because of the need to check work.
• Need open access, 'to pretty much everything', so
  that the work can be replicated and checked
  (data, methods, clear complete reports).
• (Need protocols to be published in advance.)

4
A scientific answer.

• is never something like x 1.23
• nor is it just x 1.23 0.45
• but rather also how it was derived, what
  assumptions and approximations are involved, so
  that outsiders can scrutinize.
• Just because assumptions are not mentioned does
  not mean they are not being made!

5
Some quotations

• The combination of some data and an aching
  desire for an answer does not ensure that a
  reasonable answer can be extracted from a body of
  data. John Tukey
• While every data set contains noise, some data
  sets may contain signals. Therefore before you
  can detect a signal within any given data set you
  must first filter out the noise.
• Donald J. Wheeler in Understanding Variation the
  key to managing chaos. Pub SPC

6
Time Variation in Crime

• It appears that little is known about how crime
  varies on the small scale. Therefore it is
  difficult to be clear if any changes are due to
  a crime reduction intervention.

7
The Randomised Controlled Trial(A truly
marvellous scientific invention)

• Note to avoid bias
• Allocation is best made tamper-proof.
• (e.g. use concealment)
• Use multiple blinding of
• patients,
• physicians,
• assessors,
• analysts

Population
Take Sample
Randomise to 2 groups
Old Treatment
New Treatment
Compare outcomes (averages) recognising that
these are sample results and subject to sampling
variation when applying back to the population
8
Counts of those cured and not cured under the two
treatments
Cured Not Cured
New Treatment a b
Control (Standard treatment) c d
9
The Odds Ratio
Cure Not
New a b
Old c d

• OR Ratio of the odds of cure under the two
  treatments
• Odds of cure under new treatment a/b
• Odds of cure under old treatment c/d
  
• ad
• bc
• If OR gt 1 the new treatment is better
• OR lt 1 the old treatment is better

10
But

• there is sampling variability.
• Consider a table
• 
  
• With OR 60?45 1.23
• 40?55
• is not good evidence for a difference in
  treatment effectiveness.
• The numbers are small and the sample OR 1.23
  could be due to chance, when in fact the
  population OR1.0

Cure Not
60 40
55 45
11
Sampling Variation

• Sampling Variation is given by the (asymptotic)
  standard error of ln(OR)
• (S.E (ln(OR)) )2 Var(ln(OR))
• 1 1 1 1
• a b c d
• If the events are statistically independent.
• The sample ln(OR) is distributed about the
  population ln(OR) in a Normal/Gaussian fashion
  with its standard deviation given by that
  calculated from the s.e. (5 resides outside 1.96
  s.e.)

12
Crime counts before and afterin the two areas
Before After
Treatment Area (lighting increased) a b
Comparison Area (lighting stays the same) c d

• Examine the Cross Product Ratio (CPR) a/b
• c/d
• If it is convincingly gt1 then lighting works
  against crime.

13
Lighting and crime

• Much justification for exterior lighting is made
  on the basis of crime reduction.
• (e.g. The Institution of Lighting Engineers)
• There seem to be many theoretical suggestions
  why lighting might increase or decrease crime.

14
Light has a good press!

• God said, let there be light and there was
  light. And God saw the light, that it was good.
  Genesis Chapter 1 Verses3/4.
• It seems wicked to question the benefit of
  lighting.
• However there is a dark side lightings
  environmental impacts, possible health impacts.

15
My Interest

• Paul Marchant, statistician at Leeds
  Metropolitan University who argues that
  statistics used in the Home Office Study 251
  could equally be used to show that street
  lighting actually increases levels of crime. This
  is an argument which the APPLG, alongside the
  ILE, would hope to show as utterly absurd. Of
  course it is worth noting that Paul Marchant is
  also an astronomer as well as being a
  statistician, and that this may lead to some bias
  in his interpretation of the statistics he refers
  to.
• P56 of the March/April 2004 issue of the
  Lighting Journal, the magazine of the Institution
  of Lighting Engineers.
• APPLG The All-Party Parliamentary Lighting Group
• ILE The Institution of Lighting Engineers

16
Forest Plot from Meta-analysis
Odds Ratio (95 CI)
HORS251
Study
STATA metan
Weight
17
The Confidence Intervals

• The confidence intervals of individual studies,
  which go to give the combined result, are
  calculated as though the events come from
  independent random samples.
• So that
• Var (ln(CPR)) 1 1 1 1
• a b c d
• But this can not be!

18
Correlations Within

• Crime is committed by criminals!
• Look at the data! Most studies are just counts of
  crime in the 2 areas, before and after. But one
  that is not is Bristol.
• If the independent random samples assumption were
  correct, the variance of the count would be
  expected to be approximately equal to the mean
  count. But it is not. It is an order of magnitude
  higher.

19
The Bristol Study (Shaftoe 1994)
Shaftoe said no discernable lighting benefit
but HORS251 says z6.6 ! Note had the data for
the year immediately prior to the introduction of
the relighting, i.e. periods 2 and 3, been used
rather than unnaturally using periods 1 and 2
which leaves a gap of ½ year, the effect found
would have been half of that claimed. (Shows
large variability.)
20
Overdispersion (1)

• The Bristol (and Birmingham) studies show large
  variation over time when the light level is
  constant. The variance is many times the mean
  Overdispersion.
• (The large 60 heterogeneity statistic, Q, given
  by the meta-analysis of the 13 studies, also
  suggests this. A large Q shows that there is an
  inconsistency between, within study variation and
  between study variation)

21
Overdispersion (2)

• The problem for HORS251 is that the confidence
  intervals around the effect size must therefore
  be substantially increased.
• Also because the underlying overdispersion is not
  properly known for individual studies, we can not
  say what weights we must use, as these will not
  be the same as used in the original incorrect
  HORS251 meta-analysis.

22
Examine Overdispersion in Comparison Areas
Study Name Dobs s2/?x
Atlanta 58.3594
Milwaukee 13.7547
Portland 7.7017
Kansas City 9.7449
Harrisburg 1.5748
New Orleans 46.6810
Fort Worth 0.2934
Indianapolis 0.0400
Dover 4.7647
Bristol 44.7116
Birmingham 1.5306
Dudley 4.4420
Stoke-on-Trent 0.0083
Calculated from the before and after counts in
the comparison areas
Dobs are extremely variable and right skewed. The
arithmetic mean is 15 for these comparison areas.
(Larger still if the mean includes weighting by
number of crimes.)
23
What observed overdispersions are expected if
repeated samples taken?

• It can be shown
• for a Normal(µ, s2) such that s2 ?µ, (i.e. the
  variance the overdispersion factor (k) ? the
  mean (µ) )
• that the sampling distribution of the observed
  overdispersion kobs ( s2/?x ) is (provided kltltµ)
  approximately
• Chi-squared 1df, scaled by k.
• This can be written equivalently as
• Gamma (scale 2k, shape1/2)
• This is a right skewed distribution with
  arithmetic mean k.
• If the before-after correlation ?, then k is
  replaced by k(1-?). This is the effective
  overdispersion, relevant for a before-after
  study, i.e. the quantity which is of interest.
  Indeed s2 estimates s2(1-?).

24
The CDF of the HORS251 overdispersions
k15
25
Other crime data for confirmation

• I used a data set of burglary count data from 124
  anonymised small areas. The data was from a
  project described in Tilley et al. 1999.
• Has counts of similar size to those in HORS251.

26
Burglary data from 124 areas
k10 both
27
What to conclude about overdispersion

• Both the HORS251 and the burglary data show
• great overdispersion of 10 or more.
• In the case of HORS251 and the CI is (7.9, 38.7).
• (Similar results are obtained with using the
  quasi-Poisson Generalised Linear Model in R)
• A big problem in HORS251 is essentially confusion
  about Unit of Analysis. (It is Area, not
  Crime-event)

28
The Dudley Study (1)

• Used a household crime survey. Painter and
  Farrington 1997. in Situational Crime Prevention
  Successful case studies. (Also theres the Stoke
  on Trent Study)
• Question Did you experience crime in the past
  year if so how many?
• Two areas supposedly matched one has lighting
  increased, the other stays the same. The
  household survey was carried out before and after
  the new lighting introduced. Households were
  planned to be linked before and after, but did
  not happen (nor in Stoke)!
• The reports of the studies make great claims of
  success of the effectiveness of lighting. Claimed
  by the Institution of Lighting Engineers as
  'Proof' that lighting is effective against crime.

29
The Dudley Study (2)Some Problems uncovered

• Eventually I was given some of the data. (With a
  limited number of background variables however.)
• Markedly different crime rates at the start
  between the 2 areas.
• One tailed testing used to claim a statistically
  significant effect.
• Overdispersion Variance of the number of crimes
  per household is much larger than the mean,
  therefore Poisson methods are inappropriate.

30
The Poisson Model is Inappropriate.(See below,
the distribution of crime counts for households.)
31
The Dudley Study (3)Some problems uncovered
cont.

• Differential loss to follow up.
• Old people are much less prone to experience
  crime and their number is much reduced due to
  loss to follow-up in the comparison area. So the
  relative composition changes during the
  experiment.
• Results are very sensitive to the loss or
  addition of just one person
• But importantly there is correlation between
  households, giving extra overdispersion
  (variability).
• Essentially its a non-randomised two-cluster
  trial.

32
Spatial Correlation (1)

• An expression can be derived for the variance of
• ln(CPR) for a household survey, before and
  after, intervention-comparison study, i.e. of the
  Dudley type. This includes, in addition to the
  variability between households, both
• correlations within households between times.
• correlations between households at any one time.

33
Spatial Correlation (2)

• What you get basically is the expression you
  would get if you ignored the correlation between
  households at one time, i.e. ignored the spatial
  correlation, multiplied by the Design Effect,
  Deff. (Just as in clustered surveys / trials)
• Deff(1(n-1) ?s)
• ?s the spatial correlation
• n the number in a cluster, i.e. area

34
Spatial Correlation (3)

• The spatial correlation was not taken into
  account in the Dudley and Stoke analyses thus
  ignoring the fact that neighbours share risk.
• An expression for the variance of the logarithm
  of the Cross Product Ratio CPR is

35
Spatial Correlation (4)

• ? , lambda, can be estimated from the variance
  of the number crimes experienced by households
  divided by the mean.
• r, the correlation of crimes before to after for
  the same households. This must be reconstructed
  because linking never happened. Said to be 0.3.
• But we do not know and cannot estimate Deff (Deff
  gt1) for this sort of study.

36
The response given to my pointing out that
overdispersion exists (1)

• The expression for the household survey type
  study, that I give above, but without Deff was
  used on the original Dudley Poisson result to
  give a variance adjustment of only about 3 ?
  (i.e. just ? estimate). This overdispersion
  adjustment was then applied in the meta-analysis
  for all 13 studies.
• See Addendum to HORS251 (added in Sept. 2003),
  with which I most profoundly disagree, .even
  though my name is mentioned!

37
The response given to my pointing out that
overdispersion exists (2)

• Additionally, Farrington and Welsh (2004),
  following my own short article in the BJC, cite
  the geometric mean of the s2/?x of the 13
  studies of HORS251 to justify a small value.
• However, as I have indicated here, it is the
  arithmetic mean which is appropriate, showing
  that the overdispersion is much bigger, with a
  value of something like 15.

38
The response given to my pointing out that
overdispersion exists (3)

• Farrington and Welsh justify their overdispersion
  estimate because if one divides the original
  heterogeneity statistic, Q60, by their favoured
  estimate an acceptable heterogeneity statistic
  results! Note it is usual to use Q to uncover
  anomalies in the data rather than remove them!
  The larger value of overdispersion, 15, would
  indicate excessive homogeneity Q revised 60/15
  as might result from publication bias. (There is
  no register for study protocols, which would
  guard against publication bias).

39
Lack of Equivalence between Areas

• Invariably it is the most crime-ridden area that
  gets the lighting, whereas the relatively
  crime-free control is not re-lit. So there is
  lack of equivalence at the start. One effect of
  this is to allow regression towards the mean to
  operate. (see later)
• The name Control Area is a misnomer.
  Comparison Area is a better name.

40
Line of Equality
100
Line of mean of Y for a given X
Cloud of Data Points
50
Y The after measurement
0
0
100
50
X The before measurement
41
The response given to the lack of equivalence
between the 2 areas. (RTM)

• Regression towards the mean (RTM) has not been
  acknowledged to be a problem, after I pointed it
  out.
• The burglary data shows RTM nicely. Splitting the
  data for the 124 areas into 2, above or below the
  mean burglary rate in the first year, exhibits a
  tendency in the following year for the high
  burglary rate group to show a fall and the low
  burglary rate group to show a rise.

42
RTM Example from the Burglary Data

• (Seen in period 2 to 3 also. And using rate,
  rather than count)

43
Regression Towards the Mean RTM Seeing effects
which arent there

• A statistical novice might interpret the fact
  that the high burglary rate group shows a
  reduction in burglaries (-71), as opposed to the
  low rate group (6), as evidence of something
  important going on rather than just what is
  expected when you have correlated data. RTM
  follows from correlation. (As Francis Galton
  discovered more than a century ago, in the 1880s.)

44
The response given to the lack of equivalence
between the 2 areas. (RTM)

• Farrington and Welsh (2006) claim that RTM is a
  not problem because the effect in crimes counted
  in 250 Police Basic Command Units going from
  2002/3 to 2003/4 showed only small effect. This
  is hardly surprising as the areas and hence the
  number of crimes counted are an order of
  magnitude larger than in HORS251 so the year to
  year correlation is high. Note Wrigley (1995)
  This tendency for correlation coefficients to
  increase in magnitude as the size of the areal
  unit involved increases has been known since the
  work of Gehlke and Biehl (1934).

45
Bristol study revisited

• A lighting benefit effect with p0.011 is claimed
  in reply to me. But this depends on an arbitrary,
  specific regression model that requires the
  variance to be the same in both areas and to
  include a linear time trend, identical in both
  areas, but which is not statistically
  significant.
• On the other hand, a model which just uses the
  crime count in the comparison area as predictor
  of crime in the re-lit area (in the spirit of
  HORS251) shows no stat. sig. effect.
• Does the data really look as if there is such a
  clear effect, i.e. one which would only occur 1
  time in 100, when there is in fact no lighting
  benefit?

46
The Bristol data again
It seems to me that this hardly presents clear
evidence for lighting benefit! Theres a big
problem of model uncertainty.
47
Cost benefit analyses

• Cost benefit analysis has been done based on very
  few studies by lighting and crime researchers
  (and gives a highly favourable result for
  lighting). However doing this only compounds the
  problem. As an unknown, unproven benefit/harm is
  being compared with uncertain costs.
• We need to get much better information to do such
  an exercise properly otherwise it tends to look
  scientific to the eye of a novice, when in fact
  it isnt, because of flimsy data and method.

48
Researchvertising

• Unsurprisingly HORS251 and the Dudley Study are
  used by the lighting industry to promote its
  wares. Also responsibilities under the crime and
  disorder act are invoked

49
My take on lighting and crime

• It may be that lighting reduces crime, or may be
  it increases crime. We have to look at the
  evidence as given. The conclusion, at present,
  is We do not know....yet we ought to know!
• Note, I know of no scientific trials of exterior
  'Security' lighting. So no one knows if this
  works.
• We ought to take a Popperian view and entertain
  the possibility of light being ineffective or
  worse, against crime.
• Of course we all need light at night, to see by.
  (Those concerned about light pollution are
  basically talking lamp-shades). However there
  is no sound evidence we need light to protect us
  from crime, in spite of claims.

50
Car Alarms

• It seems that there is little evidence that car
  alarms prevent cars being broken into etc.
• But they do disturb peoples sleep!
• Attempt in New York to get them banned, (rely on
  passive methods of risk reduction instead.)

51
Wider problems of inappropriate methods

• The costs of crime and attempts at its reduction
  are large.
• Similar problems probably exist for the
  evaluation of other area-based crime reduction
  interventions, too. (They certainly do for
  HORS252 on CCTV where the same methods as HORS251
  are used on 18 studies, Q270. However no effect
  of CCTV is claimed). Problems seem to be
  encouraged by the Maryland Scientific Methods
  Scale which seems to suggest that weaker
  designs, than RCTs, might suffice.
• We do need to have proper evidence to decide
  what works in crime and in all spheres .

52
Crime Prevention..an Art ?

• Or is it more like 17th Century medicine?!

53
Some considerations for evaluation of anything

• Interventions are expensive, as are the
  consequences, so their effects need to be
  researched to a very high standarde.g. the
  necessity of using randomisation.
• Also the target population could be depleted
  through poorly conducted studies.
• Evaluation needs to be done right.
• Caution is needed with systematic
  reviewing/meta-analysis as it involves moving
  away from the primary sources.
• It is possible to do roll out of a programme in
  a way that is amenable to proper scientific
  evaluation.

54
Much can be borrowed from methods in health
research

• For example in area-based crime reduction
• The Methods of Cluster Randomised Trials a
  appropriate.
• See e.g.
• Ukoumunne et al (1999)
• and
• Campbell et al (2004).CONSORT extension for
  Cluster Randomised Trials.
• Also, post-implementation surveillance is highly
  desirable for any programme.
• (Note it is problematic enough to determine What
  Works in healthcare where the unit is person,
  through e.g. dissemination bias.)

55
High standards are needed for evidence

• Need
• Adequate funding to provide quality research, as
  the costs of rolling out programmes on a national
  scale are huge. Need to be aware of the costs of
  implementing ineffective or counterproductive
  programmes.
• Effective random allocation for experiments.
• Blinding where possible, e.g. of assessors.
• Pre-publication of protocols.
• Open-access in order to check any work.
• (As, for example, reports/papers may confuse
  standard deviations and standard errors or, as
  above, not recognise correlated data.)
• Lets ultimately have the raw data!

56
Conclusion

• There is no good scientific evidence to show that
  night time lighting reduces crime. (In spite of
  the claims).
• The effect of some other crime reduction methods
  is also questionable, because of inherent
  weakness of research methods used, and needs
  examination. (Look at the variation in comparison
  areas.). Variation in crime is poorly understood.
• High quality studies, sound in design through to
  conclusion, are necessary in important and costly
  matters
• Statisticians have a vital role in finding what
  works.
• Eminence in any subject does not guarantee the
  correctness of statistical pronouncements.

57
Final points

• Do be investigative. Don't just mind your own
  business.
• --------------------------------------------------
  --------
• Stephen Senn quote Trust nobody, check
  everything from Fear and Loathing in
  Pharmaceutical Statistics, RSS2002
• HG Wells quote Statistical thinking will one day
  be as necessary for efficient citizenship as the
  ability to read and write.
• The RSS magazine Significance which aims to
  encourage statistical thinking June 2005 p62
  has an article by me Evaluating area-wide crime
  reduction measures.

58
References

• Campbell MK, Elbourne DR, Altman DG for the
  CONSORT Group (2004) CONSORT statement extension
  to cluster randomised trials. BMJ 328 702-708.
  http//bmj.bmjjournals.com/cgi/reprint/328/7441/70
  2
• Farrington D.P. and Welsh B.C. (2002) The Effects
  of Improved Street Lighting on Crime A
  Systematic Review, Home Office Research Study
  251, http//www.homeoffice.gov.uk/rds/pdfs2/hors25
  1.pdf
• Farrington D.P. and Welsh B.C. (2004) Measuring
  the Effects of Improved Street Lighting on Crime
  A reply to Dr. Marchant The British Journal of
  Criminology 44 448-467 http//bjc.oupjournals.org
  /cgi/content/abstract/44/3/448
• Farrington D.P. and Welsh B.C. (2006) How
  Important is Regression to the Mean in Area-Based
  Crime Prevention Research?, Crime Prevention and
  Community Safety 8
• Marchant P.R. (2004) A Demonstration that the
  Claim that Brighter Lighting Reduces Crime is
  Unfounded The British Journal of Criminology 44
  441-447 http//bjc.oupjournals.org/cgi/content/abs
  tract/44/3/441

59
References continued

• Marchant P.R. (2005) What Works? A Critical Note
  on the Evaluation of Crime Reduction Initiatives,
• Crime Prevention and Community Safety 7 7-13
• www.extenza-eps.com/extenza/loadHTML?objectIDValu
  e63645typeabstract
• Painter, K. and Farrington, D. P. (1997) The
  Crime Reducing Effect of Improved Street
  Lighting The Dudley Project, in R.V. Clarke ed.,
  Situational Crime Prevention Successful case
  studies 209-226 Harrow and Heston, Guilderland
  NY.
• Shaftoe, H (1994) Easton/Ashley, Bristol
  Lighting Improvements, in S. Osborn (ed.) Housing
  Safe Communities An Evaluation of Recent
  Initiatives 72-77, Safe Neighbourhoods Unit,
  London
• Tilley N., Pease K., Hough M. and Brown R. (1999)
  Burglary Prevention Early Lessons from the Crime
  Reduction Programme, Crime Reduction Research
  series Paper1 London Home Office
• Ukoumunne, O. C., Gulliford, M. C., Chinn, S.,
  Sterne, J. A. C., Burney, P. G. J., and Donner,
  A. (1999). Evaluation of Health Interventions at
  Area and Organisation level, British Medical
  Journal, 319 376-379 http//bmj.bmjjournals.com/cg
  i/content/full/319/7206/376
• Wrigley N., Revisiting the Modifiable Areal Unit
  Problem and Ecological Fallacy pp49-71 in Gould
  PR, Hoare AG and Cliff AD Eds Diffusing
  Geography Essays for Peter Haggett

60
Further material written in this area

• I wrote Failing to measure any effect of
  increased lighting on crime A reply to Profs.
  Farrington and Welsh It was sent to the Home
  Office in Dec. 2004
• www.imresearch.org/praxiscentre/Papers/RevReplyToF
  W1B.pdf
• Dr. Barry Clark of the Astronomical Society of
  Victoria, Australia has written much on the oft
  repeated but seemingly dubious claim that
  lighting reduces crime.
• http//www.asv.org.au/index.php?optioncom_content
  taskviewid33Itemid76

61
Appendix Other Useful Resources to assist in
evidence based work

• 1. RCTs designing and writing up
• The CONSORT statement (CONSORT Consolidated
  Reporting Of Randomised Trials) is very useful
  for reporting results and also thinking about
  design.
• www.consort-statement.org
• 2. The QUORUM statement Quality Reporting of
  Meta-analyses.
• 3. The American Association for Public Opinion
  Research www.aapor.org has useful information
  about Standards and Best Practices in survey
  methods