Monitoring and evaluating the impact of action on alcohol problems in Scotland

1
Monitoring and evaluating the impact of action on
alcohol problems in Scotland

• 2nd National Event for HEAT (H4) Target
• March 26th 2009
• Clare Beeston Public Health Adviser
• NHS Health Scotland Public Health Science

2
Overview

• How the evaluation is being developed
• The challenges
• How suggesting ABIs are evaluated
• Other proposed studies

3
Aims

• Track implementation progress, reach and outcomes
• Evaluate success or otherwise in achieving
  outcomes, including effects on health
  inequalities, unintended consequences and
  displacement effects
• Add to the evidence base
• Learning and improvement

4
Process

• Monitoring and Evaluation Reference Group for
  Alcohol MERGA
• Research, policy and delivery partners
• Portfolio of studies - draft
• External peer review
• Logic models to identify outcomes and develop
  evaluation framework
• Predominantly routine data robust, efficient,
  trends. Some new, some qualitative

5
Outcomes triangle - Alcohol
We live longer, healthier lives
We have tackled the significant inequalities in
Scottish society
We live our lives free from crime, disorder and
danger
We realise our full economic potential
Our children have the best start in life
NATIONAL OUTCOMES
Reduced inequalities in HLE, Improved Mental
Wellbeing, Reduced alcohol-related deaths
hospital admissions
HIGH LEVEL OUTCOMES
Less work absenteeism incapacity
Less children affected Increased educational
attainment
Reduced alcohol-related ASB crime
Reduced alcohol consumption
INTERMEDIATEOUTCOMES
Sensible drinking culture
Safer drinking wider environments
Access to affordability of alcohol
Supportive environments for children of
alcohol-abusing families
SHORT-TERM OUTCOMES
Outcomes related to service delivery
Wealthier Fairer
Smarter
Healthier
Safer Stronger
Greener
6
Evaluation Framework
Long term outcomes Changes in health, crime etc
Intermediate outcomes Changes in affordability,
attitudes, consumption, etc
Short-term outcomes Dependant on action
Outputs and reach Implemented as
intended? Proportion reached? Right people?
Monitoring Population trends (predominantly
routine)
Evaluation Focus across Interventions (routine
and new)
Monitoring implementation and Specific evaluation
7
The Challenges

• Policy/practice driven not research driven
• Attribution unravelling gossamer with boxing
  gloves
• Recession and credit crunch
• Proportionality What studies what data
• Maintaining rigor and robustness
• Managing expectations What change evaluation
  capable of detecting, what might change when
• Uncertainty about some actions

8
Evaluation of ABIs RE-AIM

• To provide a better understanding of
  effectiveness in the real world ? Better
  translation into practice need to understand
• R Reach to ultimate beneficiaries
• E Efficacy of the intervention
• A Adoption of intervention by staff
  organisations
• I Implementation of intervention
• M Maintenance by staff orgs, and by
  beneficiaries

9
The Question

• Have alcohol brief interventions been implemented
  as intended and in a way likely to make a
  difference to outcomes?

10
4 Components

• Training
• Did the training increase knowledge and
  confidence to deliver ABIs?
• Did it reach enough of the intended
  beneficiaries?
• What were the facilitators and barriers to
  implementation?
• Already commissioned and underway

11

• ii) Implementation
• How are ABIs being implemented across Scotland?
  How does it vary? Does it differ from what was
  intended?
• What are the barriers, the implications of these,
  and ways to overcome? What examples of good
  practice?
• Are enough staff delivering to make a difference?
  
• Are enough potential beneficiaries being screened
  and receiving an ABI if appropriate? Are they
  representative of those we want to reach?

12

• iii) Outcomes
• Do ABIs reduced consumption when delivered in
  real life setting?
• Control generalisation impossible -
  feasibility of follow-up and how consumption
  changes
• How is follow-up being undertaken?
• How is that working in practice?
• How does alcohol consumption change?
• Is an outcome study feasible/necessary?

13

• iv) Cost benefit
• Dependant on suitable outcome data
• What has the implementation of ABIs cost?
• What are the estimated benefits, in terms of
  numbers reached and estimated change in alcohol
  consumption, and ultimate impact on health
  outcomes, including inequalities

14
Other proposed studies

• Has the Licensing Act (and legal changes proposed
  in Changing Scotlands Relationship with Alcohol)
  been implemented as intended and in a way likely
  to impact on outcomes?
• How have self report knowledge, attitudes and
  purchasing changed?
• 3. Has consumption changed? What is the
  relationship with affordability?

15

• What is the relationship between any changes in
  consumption and specified outcomes? (Health,
  crime, child protection, education, productivity)
• What has been the economic impact on alcohol
  retailers (on off sales)
• Has the increased investment increased treatment
  and care service capacity?

16
More Information

• Clare Beeston
• Clare.beeston_at_health.scot.nhs.uk
• 0141 354 2966

17
National indicators
Improved mental wellbeing Reduced inequalities in
HLE Reduced inequalities in CHD mortality Reduced
alcohol-related hospital admissions
Delivered in partnership
Reduced alcohol consumption
Previous HEAT target
Increased awareness of/reflection on drinking
behaviour increased referrals for treatment
Reduced screening FAST positive at follow-up
Those drinking at hazardous and harmful
levels (in contact with services)
Screening Brief interventions (SIGN 74) no.
of screenings
Revised HEAT target
What the NHS has to do to reach target group and
deliver effective brief interventions in key
settings
H4Alcohol
Funding, trained workforce, evidence-based
guidance, data to manage and monitor the
delivery of interventions