Setting and Monitoring Targets

1
Setting and Monitoring Targets
Paul Fryers Doncaster Public Health Intelligence
Unit
2
National Our Healthier Nation target

• Our Healthier Nation set a national target to
  reduce deaths from circulatory diseases by 40,
  between the 1995-97 baseline and 2010
• The lines interpolating between most recent data
  and the target are presented as constant
  
  rates of change, i.e.
  reducing the
  rate by
  x per year.

3
Setting local Our Healthier Nation targets

• We calculated the 1995-97 baseline for Doncaster
  and set a local target to reduce death rates by
  40
• If everyone achieves this then the national
  target is achieved
• But inequalities remain unchanged

4
Measuring progress (comparing single years 1)

• Performance against the target is calculated as
• This ignores most of the available data on the
  rate of change
• The death rates for circulatory
  disease are falling so rapidly
  that it doesnt cause a problem
  most years but it would have given
  a false result in 1998.
• This method has been used by DoH
  
  for PIs and star ratings

B A A
5
Measuring progress (comparing single years 2)

• Performance against the target is calculated as
• This is the method used by DoH to assess PCTs
  performance in reducing teenage conception
  rates
• Once again, it ignores most of the available data
  
• Doncaster has been fortunate
  through 1998 being a very high
  year
• However we still have to explain
  why one year the rate has fallen by
  19 and two years later it has
  only fallen by 7

B A A
6
Measuring progress (comparing single years 2)

• Sheffield have not had it so lucky 1998 was
  quite a low year

7
Measuring progress (comparing single years 3)

• Performance against the target is calculated as
• where the target is one we set ourselves in
  advance
• Each PCT has set these targets in very different
  ways
• Again, this ignores most of the
  data and leaves the PCT hostage
  to fortune on year-to-year
  fluctuations.
• This method has recently been
  adopted by DoH

actual target target
8
Weaknesses of these simplistic methods

• By simply comparing two years data, or one
  years data with a target, we are ignoring most
  of the data
• We are left hostage to year-on-year fluctuations
• a huge amount of senior management time is wasted
• the public are worried unnecessarily
• we are not actually identifying the real problem
  areas

9
Measuring progress (regression)

• If we want to establish a rate of change over a
  set period of time we can fit a regression model
• If we log-transform the rates first we can fit a
  simple linear model
• The gradient for England Wales is
  0.0523 (SE 0.00148)
• The gradient for South Yorkshire is
  0.0574 (SE 0.00359)
• The two slopes are not
  significantly different (z1.332)
• Could compare observed slope with
  required target

10
Measuring progress (regression)

• Regression is appropriate if we are primarily
  interested in the whole of the period we are
  analysing.
• However, we are usually interested in our
  current rate of progress, sometimes whether we
  are on target
• Hence we are actually interested in the most
  recent part of the time series more than the
  beginning
• But we want to use all the data available to us
• We can do this using time series analysis and
  forecasting

11
Inequalities targets

• A few years after the Our Healthier Nation
  targets were set, at national level the rates are
  falling quickly
• There are reports that this is being achieved
  primarily in affluent areas the gap is
  widening
• Spearhead targets the 20 most
  deprived local authorities must
  narrow the gap between themselves
  and England Wales by at
  least 40 between 1995-97
  and 2010

12
How are Spearhead targets set?

• Simplest Assumption England Wales target is
  40
• Our target will reduce the gap between Doncaster
  and England Wales by 40 if the national target
  is exactly achieved
• This takes no account of what is
  actually happening nationally

13
We need to forecast

• Forecast what the England and Wales death rate is
  likely to be in 2010, using appropriate
  forecasting methodology
• It is clear that the England Wales rate is set
  to fall by a lot more than the 40 target
• In fact the national
  rate is
  currently
  forecast to have
  
  fallen by 56 by
  2010 from
  the
  1995-97 baseline
• Forecasting by
  Holts Method
  
  using ForecastPro
  
  software

14
Setting the local Spearhead targets

• In 1995-97 Doncasters circulatory disease DSR
  was 11.6 greater than the England Wales rate
• To reduce the gap by 40 it must only be 7.0
  above England Wales by 2010
• EW forecast to
  be 63 deaths
  per
  100,000 person-
  
  years in 2010
• Doncasters target
  DSR is 68
• This represents a
  reduction in
  death
  rates of 57

15
Monitoring progress Circulatory Disease

• Current forecasts, based on data up to 2005
  suggest that Doncaster is on target
• In fact our current forecast is that by 2010 the
  circulatory disease death rate will have fallen
  by 63 (95 CI 5570)
• Gap does appear to be
  
  narrowing (probably)
• DoH should use this
  methodology
  to
  measure progress
  
  rather than assessing
  performance using
  
  single year death
  
  rates

16
Monitoring progress All Cancers

• Identical method used to set All Cancers
  Spearhead Target target reduction is 27
• Progress less dramatic our current forecast is
  that by 2010 the circulatory disease death rate
  will have fallen by 19 (95 CI
  531)
• No evidence of gap
  narrowing
• If endpoint is to be
  the excess
  mortality
  in Spearhead LAs in
  
  2010 then the
  forecasts
  and targets
  need to be updated
  
  each year

17
Monitoring progress Teenage conception rates

• Over the past 14 years there is no evidence of
  trend
• Both Doncaster and England rates are fluctuating
  around a constant rate
• Doncasters rate is very
  significantly above that of England

18
Monitoring progress Teenage conception rates

• For Sheffield there is a hint of a rising trend,
  but not convincingly different from the national
  flat trajectory

19
LAA within LA inequalities targets

• Spearhead targets are a national attempt to
  measure whether the gap is narrowing
• The danger is that we can achieve our Spearhead
  target by targeting the more affluent population
  in Doncaster, so we actually widen the gap within
  Doncaster and others
• We can try to measure the gap, but its very
  difficult
• Local Area Agreements have set us a target to
  narrow the gap in all cause all age mortality
  rates between the most deprived quintile of areas
  in the borough and the most affluent quintile

20
Can we tell whether we are narrowing the gap?

• Doncaster has a population of about 290,000
• However they are defined, the most deprived and
  most affluent quintiles are going to have
  populations of about 60,000
• This makes them highly subject to random
  year-to-year fluctuations in death
  rates

• Graph shows that any
  slight reductions in
  
  the gap will be over-
  whelmed by huge
  
  fluctuations

21
Can we tell whether we are narrowing the gap?

• Every year the standardised death rates in the
  most deprived quintile have been between 30 and
  65 higher than those in the most affluent
  quintile, with no sign of a trend
• We could set a target to reduce this excess by
  20 by 2010 from a 1995-97 baseline
• The excess in 1995-97
  was 44 in 2002-04
  
  it was 50
• It should be clear by
  2010 if there has
  
  been progress but it
  will be
  impossible to
  measure progress on
  an annual
  basis

22
Summary

• Its very difficult to detect differences between
  rates of change, when both series are heading
  generally in the same direction
• Almost all health outcomes we monitor fluctuate
  from year to year at local level we have to see
  through these
• The way central bodies tend to monitor us tends
  to be extremely crude and very misleading
• The most useful methodology is Holts method, for
  forecasting whether we are likely to hit targets,
  and for giving current estimates of rates of
  change while using all the available data
• Targets to narrow the gap within local
  authority areas are good for raising the profile
  of inequalities, but care must be taken in
  interpreting them

23
Thank you
www.doncasterhealth.co.uk/phiu