Evaluating Nutrition Education Programs - PowerPoint PPT Presentation

Loading...

PPT – Evaluating Nutrition Education Programs PowerPoint presentation | free to view - id: 24d596-ZDc1Z



Loading


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation
Title:

Evaluating Nutrition Education Programs

Description:

Outcome evaluation as defined by USDA Food Stamp Nutrition Education Guidance (March 2007) ... experiences in teaching nutrition education in the classroom. ... – PowerPoint PPT presentation

Number of Views:47
Avg rating:3.0/5.0
Slides: 98
Provided by: lesliel150
Category:

less

Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: Evaluating Nutrition Education Programs


1
Evaluating Nutrition Education Programs
  • Leslie A Lytle, PhD, RD
  • Professor
  • Division of Epidemiology and Community Health
  • University of Minnesota

2
l
  • Lytle_at_epi.umn.edu
  • Cyhp_at_epi.umn.edu (Center for Youth Health
    Promotion website at UMN- tools, materials
    available for free)

3
The WINNERS!!!
  • How to assess behavior
  • Evaluating young children
  • Assessing knowledge and attitudes
  • Validity and reliability
  • Sample size (unit of assignment)
  • Strong study design

4
Other questions
  • How to evaluate one-shot nutrition ed?/minumum
    exposure for nutr.ed to stick?
  • Evaluating parental response to interventions
    targeted at the child (TEENS, CATCH examples)
  • Home influence on kids eating choices
  • Teacher evaluating change in kids

5
Types of evaluation
  • Outcome evaluation as defined by USDA Food Stamp
    Nutrition Education Guidance (March 2007)
  • Outcome- which demonstrates changes that occur
    in the presence of an intervention but do not
    establish cause and effect conclusions.

6
Types of evaluation
  • Impact evaluation as defined by USDA Food Stamp
    Nutrition Education Guidance (March 2007)
  • Impact-which indicate how effective the
    intervention was in changing the target
    populations attitude, awareness and/or behavior.
    (knowledge, skills, attitudes, intention to act,
    behavior, or something else (pg 29 FTNEPG,
    3/2007)
  • Assumes that a causal relationship is being tested

7
Assessing behaviors, knowledge and attitudes
  • What behaviors to focus on?
  • How do knowledge and attitudes fit in?
  • What tools are out there?
  • Which behaviors are easier/more difficult to
    assess?
  • How does age influence the evaluation tools that
    we use?

8
Recommended Behavioral Outcomes from FNS
  • Eat fruits and vegetables
  • Eat whole grains
  • Consume fat-free or low-fat milk products every
    day
  • Be physically active every day
  • Balance caloric intake from food and beverages
    with calories expended

9
Recommended Behavioral Outcomes from FNS
  • Evaluation tools should reflect knowledge,
    attitudes, behaviors related to these behavioral
    outcomes

10
Designing an intervention and evaluation plan
  • START with a conceptual model or logic model

11
Theory of Planned Behavior
Attitude toward the behavior
Subjective norm
Perceived behavioral control
12
Examining nutrition education impact/outcomes
  • KNOWLEDGE
  • ?
  • ATTITUDES
  • ?
  • BEHAVIORS

13
Example of how KAB should line up in evaluation
  • KNOWLEDGE Does the student know how long is
    needed for adequate handwashing?
  • ATTITUDES Does the student believe that it is
    important to wash their hands?
  • BEHAVIORS Does the student report frequent
    handwashing or is observed consistently washing
    hands?

14
Examining nutrition education impact/outcomes
  • KNOWLEDGE
  • Why versus How to knowledge
  • How to KNOWLEDGE
  • Knowledge that will help one make healthier
    choices rather than knowledge for the sake of
    knowledge

15
Examining nutrition education impact/outcomes
  • WHY KNOWLEDGE
  • Which of the following organs is responsible
    for secreting insulin?
  • a. Kidneys
  • b. Lungs
  • c. Liver
  • d. Pancreas

16
Examining nutrition education impact/outcomes
  • HOW TO KNOWLEDGE (Example from Univ. of
    Nebraska)
  • Which food is a lower fat snack?
  • a. Pretzels
  • b. Potato chips
  • c. Doughnuts

17
Examining nutrition education impact/outcomes
  • HOW TO KNOWLEDGE (Example from Univ. of Wyoming)
  • Write the name of one food found in each food
    group from MyPyramid.
  • Milk group
  • Grain group
  • Meat group
  • Fruit group
  • Vegetable group

18
More how to knowledge questions(Kansas State)
  • What is the healthiest snack choice?
  • Soda pop and chips
  • Milkshake and fries
  • Fruit juice and pretzels
  • Which foods would always be safe to pack in a
    sack lunch?
  • Sliced ham
  • Peanut Butter
  • Sliced cheese
  • How long should I wash my hands before I touch or
    eat food?
  • As long as it takes to say my name
  • As long as it takes to sing Happy Birthday
  • As long as it takes to eat an apple

19
Assessing attitudes
  • Motivation
  • Beliefs, opinions
  • Perceived benefits/barriers
  • Subjective norm (how do others feel about this
    behavior?)
  • Self efficacy (how confident do I feel about
    performing some behavior?

20
Attitude examples (Iowa State)
  • I like to eat fruits and vegetables for snacks
  • My friends like to eat fruits and vegetables for
    snacks
  • How sure are you that you can fix fruits and
    vegetables to eat as snacks at home?

21
(No Transcript)
22
Measuring behavior
  • Preference
  • Self-report behavior
  • Observed behavior

23
(No Transcript)
24
Other preference type questions
  • See CATCH HBS Section A What food would you
    pick?
  • Fruit and veggies lists
  • 5-a-day Card Sorting Task for younger kids

25
Behavioral measures to assess
  • Eat fruits and vegetables
  • Eat whole grains
  • Consume fat-free or low-fat milk products every
    day
  • Be physically active every day
  • Balance caloric intake from food and beverages
    with calories expended

26
Eat Fruits and Vegetables Self report measures
  • 24 hour recalls (valid in youth as young as 8
    years old)
  • Youth Adolescent Questionnaire-YAQ food frequency
    (valid in youth from about 6th grade and up)
  • Validated 6 item fruit and veggie screener from
    BRFSS (valid in youth from about 6th grade and up)

27
Assessing fruit and veggie intake 6th graders
and olderBRFSS tool
  • These questions are about the foods you usually
    eat or drink. Please tell us how often, over the
    past year, you ate or drank each one. Mark the
    box for the response that best describes how
    often you eat or drink the food. Be sure to
    include foods you ate or drank at home, at
    school, at restaurants or anywhere else.
  • 1. Over the past year, how many times did you
    drink fruit juices such as orange, grapefruit, or
    tomato? (Mark only ONE box.)

oNever or less than once per MONTH o1 time per DAY
o1-3 times per MONTH o2 times per DAY
o1-2 times per WEEK o3 times per DAY
o3-4 times per WEEK o4 times per DAY
o5-6 times per WEEK o5 or more times per DAY
28
  • 2. Over the past year, how many times did you eat
    fruit (not counting juice)? Mark only ONE box.)

oNever or less than once per MONTH o1 time per DAY
o1-3 times per MONTH o2 times per DAY
o1-2 times per WEEK o3 times per DAY
o3-4 times per WEEK o4 times per DAY
o5-6 times per WEEK o5 or more times per DAY
29
  • Over the past year, how many times did you eat
    green salad? (Mark only one box.)

30
  • Over the past year, how many times did you eat
    potatoes (not including french fries, fried
    potatoes, or potato chips)?

31
  • Over the past year, how many times did you eat
    carrots? (Mark only ONE box.)

32
  • Over the past year, how many times did you eat
    vegetables (not counting carrots, potatoes, or
    salad)? (Mark only ONE box)

33
Eat Fruits and Vegetables Self report measures
  • How about for kids younger than 8?
  • Observation
  • Parental reporting

34
Behavioral outcomes to assess
  • Fruits and vegetables
  • Whole grain consumption Are there valid and
    reliable tools? Can youth identify whole grain
    foods?
  • Fat free or low fat dairy
  • Be physically active every day
  • Balance caloric intake from food and beverages
    with calories expended

35
Behavioral outcomes to assess
  • Fruits and vegetables
  • Whole grain consumption
  • Fat free or low fat dairy EAT Survey 10-item
    scale
  • Be physically active every day
  • Balance caloric intake from food and beverages
    with calories expended

36
Assessing Calcium intakes in middle school age
youth
  • Project EAT (Lisa Harnack, P.I.)
  • 10 item scale
  • Reliability and validity assessed
  • Reliability testretest one week apart
  • Validity 3 24-hour recalls as criterion measure
  • Harnack, Lytle, Story et al, JADA 2006106
    1790-1795

37
Reliability and Validity
  • Reliability will the instrument produce the
    same result if applied twice?
  • Do respondents understand the question in a
    consistent manner? (test-retest)
  • Are observers consistent in documenting what they
    see? (inter-rater reliability)
  • Do the items that I have included to assess a
    construct relate to each other as I would expect
    them to? (cronbachs alpha)

38
Reliability and Validity
  • Criterion Validity How well does a measure
    compare with a gold standard or criterion
    measure?
  • Face Validity Would others reading the item
    understand the concept that I am trying to
    assess? Does the measure appearto measure the
    desired construct?
  • Content validity Does my instrument capture all
    the elements that I think are important
  • Construct validity Is the measure related to
    other measures as one would expect?

39
Assessing Calcium intakes in middle school age
youth
  • Test-retest correlations Reliability
  • Sample ICC
  • Total sample 0.74
  • Females 0.79
  • Males 0.66
  • 11-12 y.o 0.74
  • 13-14 y.o. 0.75

40
Assessing Calcium intakes in middle school age
youth
  • Assessing Validity Comparing calcium intake
    using 3 methods (n248)
  • 24 hour recalls 993mg (499)
  • 10 item CA FFQ 856mg (570)
  • 1 YRBS question 423mg (344)
  • Correlations
  • Recalls and FFQ 0.43
  • Recalls and YRBS 0.37

41
Behavioral outcomes to assess
  • Fruits and vegetables
  • Whole grain consumption
  • Fat free or low fat dairy
  • Be physically active every day
  • Balance caloric intake from food and beverages
    with calories expended

42
TEENS Survey Questions
  • Do you get some regular physical activity outside
    of school? By regular we mean at least three
    times a week for at least 20 minutes at a time.
  • Most of the time
  • Usually
  • Once in a while
  • Hardly ever
  • Never
  • When you think about how hard you work out when
    you are physically active, do you find that you
    are
  • Breathing much harder than usual
  • Breathing somewhat harder than usual
  • Breathing only a little harder than usual
  • Breathing the same as usual

43
  • How many hours per day do you usually watch TV
    during the weekdays?
  • I dont watch TV during the weekdays
  • Less than 1 hour per day
  • 1-2 hours per day
  • 3-4 hours per day
  • More than 4 hours per day
  • How many hours per day do you usually watch TV
    during the weekend?
  • I dont watch TV during the weekend
  • Less than 1 hour per day
  • 1-2 hours per day
  • 3-4 hours per day
  • More than 4 hours per day

44
  • How many hours per day do you usually play video
    games (including hand-held video games and
    computer games) during the weekdays?
  • I dont play video games during the weekdays
  • Less than 1 hour per day
  • 1-2 hours per day
  • 3-4 hours per day
  • More than 4 hours per day
  • How many hours per day do you usually play video
    games (including hand-held video games and
    computer games) during the weekend?
  • I dont play video games during the weekend
  • Less than 1 hour per day
  • 1-2 hours per day
  • 3-4 hours per day
  • More than 4 hours per day

45
Behavioral outcomes to assess
  • Fruits and vegetables
  • Whole grain consumption
  • Fat free or low fat dairy
  • Be physically active every day
  • Balance caloric intake from food and beverages
    with calories expended

46
Balance caloric intake from food and beverages
with calories expended
  • Assessing caloric intake and expenditure is very
    difficult
  • Related behavioral messages and outcomes to
    assess on the energy intake side may include
  • Consumption of a healthy portion size
  • Eating meals and few snacks
  • Conscious eating
  • Reduction of empty calories
  • Reduction of soft drinks

47
Balance caloric intake from food and beverages
with calories expended
  • Assessing caloric intake and expenditure is very
    difficult
  • Related behavioral messages and outcomes to
    assess on the energy expenditure side include
  • Reducing television time
  • Increased walking or biking for transportation
  • Small increases in activity every day
  • More leisure time activity

48
Balance caloric intake from food and beverages
with calories expended Examples
  • On a typical day, how many hours do you watch TV,
    view videos or work/play on the computer? (Iowa)
  • I dont watch tv, etc..
  • Less than one hour
  • 1 hour/day
  • 2 hours/day
  • I eat breakfast in the morning (UMN)
  • Never Sometimes Almost Always Always

49
Energy balance knowledge Examples from IDEA
  • Question correct
  • Youth Parent
  • If someone sits all day, they do
  • Not need to eat any calories (F) 78 98
  • Alcohol contains calories (T) 77 99
  • The sweetner used in Gatorade and
  • Other soft drinks is healthier than the
  • Sweetener used in sodas (F) 47 69
  • Most youth who are active need to
  • Consume sports drinks to replace
  • Electrolytes and minerals (F) 40 72

50
How do assess young children (less than 7 years
old or 2nd grade and younger)
  • Very challenging
  • Cannot think abstractly, limited reading and
    writing skills
  • Will be able to perform simple knowledge tests
  • Use observation assessment tools as possible,
    conducted by objective observers, to assess
    behavior

51
Consider outcomes that do not rely on self report
  • Plate waste (Colorado)
  • Visually Monitoring Consumption
  • Take versus Eat

52
Visually monitoring consumption
  • Purpose To develop and evaluate a simple
    observation method to assess student consumption
    of specific menu items offered in school
    cafeterias
  • Developed the Visual Food Monitoring Form
  • Selected 4 menu items to estimate consumption
  • End of tray line, dots on trays with items
    (established our denominator)
  • At tray return area, staff completed the VFMF
  • Snyder, Fee, Lytle, Hann, 1996 School Food
    Service Research Review, 20(2), 63-68.

53
(No Transcript)
54
Results
  • Evaluation staff made 3 visits to 4 schools
  • Observed 1839 trays
  • Conducted pre-post evaluations of modified menu
    items
  • Results
  • 81 ate at least 3/4 of higher-fat entrées
  • 78 ate at least 3/4 or lower-fat modifications

55
Can food taken be used as a proxy for food eaten?
  • Purpose If we record what foods are on
    students trays at the end of the cafeteria line,
    can that be used as a proxy for foods consumed?
  • Rationale Observing and recording actual
    consumption is labor-intensive and expensive,
    (approximately 40/child observed!)
  • Gray, Lytle, Mays, et al, 2002 JADA 102407-409.

56
Can food taken be used as a proxy for food eaten?
  • Validation study n350 5th grade students
  • Recorded foods present on trays at end of tray
    line
  • Recorded foods actually consumed
  • Used different staff to record taken and eaten
    data.
  • Results
  • Correlations of takeneaten fruits 0.59
  • Correlations of taken eaten veggies0.51
  • Gray, Lytle, Mays, et al, 2002 JADA 102407-409.

57
(No Transcript)
58
Types of evaluation
  • Outcome evaluation as defined by USDA Food Stamp
    Nutrition Education Guidance (March 2007)
  • Outcome- which demonstrates changes that occur
    in the presence of an intervention but do not
    establish cause and effect conclusions.
  • Study design Post-only with a
    control/comparison group
  • Pre-post

59
Pre-post study design
  • What are some of the problems with this design?

60
Pre-post study design
  • What are some of the problems with this design?
  • Was the change REALLY caused by the intervention?
  • Some other thing happened?
  • Response bias? Did they learn the right answer?

61
Pre-post study design
  • What are some options?
  • Find a comparison/control group
  • Do a delayed intervention condition
  • Use different interventions in different classes
    and use one evaluation tool that assesses
    behavioral objectives from both intervention

62
  • Do you need to conduct an evaluation every time
    you run an intervention?

63
Sample study design
10 schools post-treament
10 schools intervention
1st follow up eval
  • Baseline
  • measures
  • 20 Schools

Final follow-up measures
R
10 school waiting list
1st follow up eval
10 school intervention
1/2 I
M
M
1/2 I
M
Rrandomization
Mmeasurement period
64
Impact evaluation
  • Impact evaluation as defined by USDA Food Stamp
    Nutrition Education Guidance (March 2007)
  • Impact-which indicate how effective the
    intervention was in changing the target
    populations attitude, awareness and/or behavior.
    (knowledge, skills, attitudes, intention to act,
    behavior, or something else (pg 29 FTNEPG,
    3/2007)
  • Assumes that a causal relationship is being
    tested
  • Study design Randomized control/comparison
    design

65
Establishing a causal relationship
  • What is meant by a causal relationship?
  • A --- B
  • If we introduce A, then B is more likely to occur
    than if A was NOT introduced
  • B doesnt ALWAYS occur with A
  • Sometimes B occurs even without A
  • There is some assumption of temporality (A
    precedes B)

66
What are the best study designs to evaluate
IMPACT?
  • Have at least 2 time points
  • Clearly identify what A is
  • Clearly identify what B is
  • Anticipate potential confounding factors and
    assess them
  • Have a control or comparison group
  • Have an adequate sample

67
Control versus Comparison group
  • Control groups are randomized
  • Comparison groups are NOT randomized

68
Gold standard for assessing impact
  • Randomized Control Trial
  • Unit of assignment is unit of randomization
  • Clustering that naturally occurs in groups
    included in the analysis plan (random effects)
  • Sample size calculations are based on the unit of
    assignment/randomization
  • Baseline measures, randomization, intervention,
    follow-up measures
  • Can be very expensive

69
  • Randomized control trial with attention paid to
    unit of assignment
  • What is Unit of Assignment?

70
Unit of assignment and the FNS document
  • Page 7
  • Describe the unit of assignment to intervention
    and control groups
  • Describe how assignment to intervention and
    control groups was carried out
  • Describe how many units and individuals were in
    the intervention and control groups at the start
    of the intervention.

71
Unit of Assignment
  • At what UNIT is the intervention delivered? At
    what UNIT does randomization into condition
    occur?
  • Out-patient nutrition counseling Individual
  • Community trial An entire community
  • FNS nutrition programs A classroom, a school, a
    school district, an afterschool program

72
What is the Unit of Assignment when you
  • Train all teachers in a school to deliver an
    intervention?
  • Work with the school food service director at the
    district office to change offerings in school
    cafeterias?
  • Recruit youth in an after school program to
    participate in a special nutrition education class

73
TEENS as an example
  • 16 schools were randomized into control or
    treatment condition
  • School is the unit of assignment and analysis
  • Over 3500 students in 16 schools completed
    surveys
  • The sample size of TEENS 16

74
Unit of assignment
  • Consider a classroom or a school as the unit of
    assignment
  • One classroom/school receives an intervention
  • One classroom/school does not
  • The mean is the aggregated scores of all kids in
    the classroom/school
  • The analysis compares the means between the units
    of assignment

75
  • School or classroom becomes the unity of
    assignment and analysis
  • May be able to add other elements of the
    intervention to the model
  • The analysis model needs to include a term for
    the clustering of kids within the unit

76
  • What are some potential problems in using a
    classroom as the unit of assignment and then
    comparing students in control and
    intervention classrooms?

77
Keep in mind
  • Focus on specific behaviors
  • Attempt to focus your intervention (content,
    time, type of delivery)- too much flexibility is
    very difficult to evaluate
  • Make sure your intervention objectives,
    intervention strategies, and KAB measures are all
    lined up

78
Keep in mind
  • Look for valid and reliable instruments in the
    literature
  • Observation may be a cost-effective way to assess
    change-especially with young kids
  • Keep it simple
  • Move beyond pre-post study designs
  • Look for a control/comparison group

79

80
Why evaluate?
  • Fulfill an obligation or regulatory requirement
  • Document how your program is being implemented
  • Document how your program is being received
  • Document if your program is working
  • Provide feedback for future program planning

81
Types of evaluation Qualitative and quantitative
evaluation
  • Qualitative
  • Open ended questions
  • Focus groups/key informant interviews/observations
  • Allows for stories to be told
  • Allows for respondents to describe their own
    experiences or opinions

82
Types of evaluation Qualitative and quantitative
evaluation
  • Quantitative
  • Closed ended questions
  • Response can be QUANTIFIED/COUNTED
  • Surveys/questionnaires/observations
  • Allows for presentation of data in a tabular
    format
  • Allows for statistical comparison
  • Respondents provide responses to specific
    questions

83
Types of evaluation
  • Qualitative EXAMPLE
  • Interview teachers prior to the development of a
    new set of classroom materials. Ask them the
    following question
  • Tell me about your experiences in teaching
    nutrition education in the classroom. What are
    some things that you enjoy about teaching
    nutrition education to your children? What are
    some things that make it a challenge?

84
Types of evaluation
  • Quantitative EXAMPLE
  • Ask teachers to fill out a survey that includes
    the following question
  • Which of the following make teaching nutrition
    education challenging? Check all that apply
  • Not enough classroom time
  • Available materials are not culturally
    appropriate
  • Available materials are not age appropriate
  • No available materials
  • Lack of support for nutrition education from
    administration
  • Lack of cooking/food preparation facilities
  • Lack of interest by students
  • Inadequate background in nutrition

85
  • When might you use QUALITATIVE EVALUATION?
  • When might you use QUANTITATIVE EVALUATION?

86
Types of evaluation
  • Formative
  • Process
  • Impact
  • Outcome

87
Types of evaluation
  • Formative Conducting information for the intent
    of FORMING or developing a program or for FORMING
    or adapting a program

88
Process Evaluation
  • Part of formative evaluation
  • Provides documentation of what is going on in a
    program, how a program is being implemented and
    received by a variety of stakeholders

89
Formative evaluation Examples
  • Conducting focus groups with teachers and youth
    to see what types of nutrition programs that they
    are interested in
  • Conducting a survey to document parents beliefs
    about the school food environment
  • Conducting key informant interviews after the
    completion of a pilot program to see what
    teachers think needs to be changed for the next
    phase of programming

90
Process Evaluation
  • May provide documentation of
  • Completeness of implementation Is all of the
    intervention being implemented as planned?
  • Fidelity to the intervention plan Is the
    intervention being implemented as planned?
  • Dose of the intervention How much of the
    intervention is being received by intended
    parties?
  • Reach of the intervention How many of the
    intended participants are receiving the program?
  • How do stakeholders feel about the program?

91
Kansas StateProcess Evaluation Examples
  • Do you plan to spend more time on nutrition
    education in your classroom than you did prior to
    this nutrition education program?
  • How was class response to the program?
  • What part(s) of the program did students enjoy
    the most?
  • Would you be interested in having the program
    again next year?
  • Do you have any requests, suggestions, ideas for
    improvements, or other comments?

92
(No Transcript)
93
(No Transcript)
94
(No Transcript)
95
  • But isnt it all about the environment?
  • Do we have a chance to positively influence
    behaviors when the environment is toxic?

96
An Ecological Model
Davison-Birch, 2001
97
(No Transcript)
About PowerShow.com