National Obesity Observatory The Standard Evaluation Framework Kath Roberts [e-mail]

1
National Obesity Observatory The Standard
Evaluation Framework Kath Robertskatharine.ro
berts_at_empho.nhs.uk
2
What is evaluationbring me the evidence!
3
One question..
If evaluation was an animal what would it be?
4
Scary?
5
Tricky to get hold of
6
Prickly?
OTHERS
7
Has it achieved its objectives?
Has it worked?
Does it work?
What happened?
Was it effective?
Project
Do I need a double blind randomised control
trial?
Could it work?
What questionnaire should I use?
8
Standard Evaluation Framework (SEF) for weight
management interventionswww.noo.org.uk/SEF
9
Aim of the SEF

• To support high quality, consistent evaluation of
  weight management interventions in order to
  increase the evidence base

10
Objectives of the SEF

• Increase the proportion of weight management
  interventions that are evaluated
• Increase the quality of evaluations
• Develop a core dataset to increase the
  consistency and comparability of evaluations
• Provide wider evaluation support

11
Target audience for the SEF

• PCT commissioners (to support service
  specifications)
• PCT DH obesity leads
• Organisations running interventions
• External evaluators
• The NE region requires the SEF is incorporated
  as an essential component of all new weight
  management programmes?

12
Why should we use the SEF?

Evidence-based interventions
Number of interventions
Speculative interventions
Time
13
What interventions should it apply to?

• Any intervention that explicitly sets out to
  manage or reduce body weight as an intermediate
  or long-term health outcome (including the
  primary prevention of weight gain)
• Projects focusing on diet, physical activity, or
  both in combination
• Intended to be applicable to a range of
  approaches including interventions conducted with
  individuals on a one-to-one basis, or with people
  in groups and in clinical or community settings.

14
The SEF is not intended for

• Medical treatment e.g. bariatric surgery
  prescribing
• Wider environmental interventions e.g. changes to
  the built environment (although the SEF could
  support the evaluation of such interventions
  alongside wider evaluation criteria)

15
Consultation Processes

• Developed in consultation with academic experts,
  representatives from Public Health Observatories,
  Primary Care Trusts, Government Offices, Local
  Authorities and other relevant organisations and
  bodies
• Initial list of criteria consulted on using an
  online questionnaire - followed up by individual
  interviews
• Two consultation meetings to finalise criteria
  and guidance.
• NB the SEF itself will also be evaluated.

16
What is in the SEF document?

• An introduction to the concepts of evaluation
• A list of data collection criteria divided into
  essential and desirable
• A guide to using the criteria and collecting the
  data

17
Introduction to Evaluation

• What is evaluation and why do it?
• What are the different types of evaluation?
• What are the differences between evaluation,
  audit and monitoring?
• What sorts of information need to be collected to
  carry out evaluations?
• What sorts of issues need to be considered as
  part of an evaluation?

18
SEF Criteria

1. Programme details e.g. name of intervention,
   aims/objectives
2. Demographics of individual participants e.g. age,
   sex, ethnicity
3. Baseline data e.g. height/weight, dietary and
   physical activity behaviours
4. Follow-up data (impact and process evaluation)
5. Analysis and interpretation

19
Supporting SEF guidance

• Details about why criteria have been categorised
  as essential or desirable
• What the data can and cannot tell you
• Standardised methods for collecting some of the
  data
• Suggested standardised definitions for data
• Suggested data collection tools
• Signposting to further guidance and useful
  information

20
Recommended Physical Activity Measurement Tools

• 4 instruments met the following criteria
• Relatively easy and practical to administer
• Mean difference between energy estimated by
  self-report and doubly labelled water lt40
• Instrument should have been used in a trial of
  exercise promotion and detected a significant
  difference between intervention and control
  participants
• Population in the validity study generalisable to
  wider population
• Appendix to SEF by Dr Melvyn Hillsdon

21
Recommended Physical Activity Measurement Tools

• International Physical Activity Questionnaire
  (IPAQ) long version
• 7day recall
• Valid in 15-69 year olds only
• Stanford 7-day recall
• Self report for sleep, moderate, hard and very
  hard activities
• Valid in adults only

22
Recommended Physical Activity Measurement Tools

• 7-day physical activity diary
• Self-completion diary ticking 15min blocks of
  activity
• Valid in adults only
• New Zealand Physical Activity Questionnaire
• 7day Self-completion recall, using prompt cards
• Valid in middle aged adults only

23
Limitations

• No existing physical activity measure perfectly
  meets all criteria
• Lack of suitable measures for children
• No gold standard for measuring sedentary
  behaviours
• Most measures have quite complex scoring systems

24
Future SEF products

• Stand alone evaluation guide
• Stand alone criteria tables
• Example case studies
• Excel template for audit data
• Excel template for individual study data
• Online audit (currently taking place in the NE),
  national audit under development.
• Dietary tool guide
• Commissioners guide to weight management
  interventions
• Others

25
Questions?

• Queries to
• info_at_noo.org.uk
• PHINE SEF support group
• http//www.phine.org.uk/group.php?gid99

26
Interactive Workshop

• Aims
• Share how you plan to use the SEF in your
  locality.
• Discuss using the SEF for an intervention(s)
  running or due to run in your locality and
  identify
• Possible barriers/difficulties to using the SEF.
• Possible solutions to these barriers and how NOO
  can best support these.
• The advantages the SEF would bring to this
  intervention.

27
Workshop Feedback

• Each group to report back
• How is the SEF going to be used in your locality?
• What possible barriers are there to using the
  SEF?
• How could these barriers be overcome?
• What could NOO provide to support use of the SEF
  further?