Can a Very Large-Scale, Diverse Nutrition Initiative Be Evaluated at Multiple Levels of Impact? B Foerster1, H Chipman2, J Guthrie3, C Olander4, Eileen Stommes3, S Nitzke5, J Voichick5 1 California Dept. of Health Services; 2 South Dakota State

1
Can a Very Large-Scale, Diverse Nutrition
InitiativeBe Evaluated at Multiple Levels of
Impact?B Foerster1, H Chipman2, J Guthrie3, C
Olander4, Eileen Stommes3, S Nitzke5, J
Voichick51 California Dept. of Health
Services 2 South Dakota State University and
CSREES, USDA 3Economic Research Service, USDA
4Food and Nutrition Service, USDA 5Cooperative
Extension, University of Wisconsin

• American Public Health Association
• November 19, 2003 San Francisco, CA

2
Objectives

• Introduce issues in evaluation of state Food
  Stamp Nutrition Education
• Describe evolution of state and national
  evaluation efforts
• Synthesize the issues as a public health
  nutrition practice challenge, from a state
  perspective

3
What Is Food Stamp Nutrition Ed (FSNE)?

• Optional administrative activity for Food Stamp
  state agencies
• Nut Ed any combination of activities leading
  to the voluntary adoption of healthy eating
  behaviors
• Cooperative Extension started in early 1980s
  social marketing in mid-1990s
• 5050 Federal Financial Participation with USDA
  in-kind OK
• Grew from 0 to 49 states lt1 Million to 250
  Million by FFY 2003
• Federal oversight Annual Guidance from HQ state
  plans and semi-annual reports approved by FNS
  Regional Offices

4
Association of State Nutrition Network
Administrators(Formal Social Marketing Nutrition
Networks)

• Alabama Missouri
• Arizona Nevada
• California New Jersey
• Colorado North Carolina
• Georgia Oklahoma
• Iowa Oregon
• Kansas Pennsylvania
• Maine South Dakota
• Michigan Washington
• Minnesota Wisconsin

5
Three USDA Agencies Participating Other
OversightFrom a State Perspective

• Food and Nutrition ServiceFood Stamp Program,
  the Office of Analysis, Nutrition and Evaluation
  8 Regional Offices, Internal Audits
• Economic Research Service, Food Assistance and
  Nutrition Research Program
• CSREES
• Office of Management and Budget
• Government Accounting Office
• Office of the Inspector General?
• State agenciesFood Stamp health
  department/university

6
Benchmark Recommended Priorities w/ Next Steps,
Mid-late 90sWhite Papers Published in JNE
Supplement

• Diet quality
• Food insecurity
• Food safety
• Food resource management
• Systems and environmental change

Papers Soon To Be Available on ERS Website
www.ers.usda.gov
Papers Soon To Be Available on ERS Website
www.ers.usda.gov
7
Benchmark ERS Workshop, January 2003

• Small group format (USDA agencies, other feds,
  academics, state agency)
• Identify research needs to support evaluation
• Focifurther define outcomes develop info
  systems, process measures, strategies for
  collaboration maintain ongoing dialog and
  evaluation research processes
• ERS FSNE ExtraNet, FNE Conference and Web Page w/
  data links

8
Benchmark SNE/ERS Conference, July 2003
Objectives

• Identify potential outcome measures for nutrition
  education w/ low income consumers
• Explore automated, web-based systems for
  evaluation and linking process strategies
• Identify how program reporting systems can be
  supported and process strategies linked
• www.ers.usda.gov (topics include diet and health,
  food security, links, data base of projects)

9
SNE/ERS Highlights Challenges in Outcome
Measurement(Isobel Contento, PhD)

• Stakeholders, partners, communities
  accountability, effectiveness
• Deciding on objectives, core elements diversity
  and creativity of programs general planning
  framework (What to put in the impacts box?)
• Measures for individual factors (validity and
  reliability of tools)
• Defining measures for environmental factors
  (organizational community social structure,
  policy and systems)
• Measuring impacts of environmental change
  (institutional, community, state levels)

10
SNE/ERS Highlights Recommendations for Outcome
Measurement(Isobel Contento, PhD)

• Strengthen w/ state and national evidence (beyond
  local) decide on core behaviors and impacts for
  all programs
• Match intervention objectives w/ measurement
  tools make appropriate for duration and
  intensity
• Use control/comparison groups
• Monitor environmental noise and mediating
  variables
• Use appropriate theories to select measures
• Track over time find creative ways of capturing
  data, e.g., on-line reporting, partnering

11
SNE/ERS Highlights Existing State FSNE
Reporting SystemsResults of a Survey, Spring 03
(n 42 states)

• Both quantitative and qualitative data are
  collected by virtually all S/A on 4 core areas
  of those, food security is least reported
• There is a reporting pyramid by Level of
  Influence, e.g. 46-88 of S/A collect data _at_
  Individual/Household level v. 7-17 _at_ Social
  Structure/Policy Change
• At the Community level, formal partnerships were
  the most common measures
• At the Social and Policy levels, fewer than 4
  programs reported on public opinion or public
  policy
• These data should be mined further

12
SNE/ERS Highlights Issues in Automated Data
Systems

• Other agencies are doing iteducation, substance
  abuse, social science
• Why? To link activity w/ performance, connect
  groups, minimize reporting burden
• Many technical considerations in designing
  systems
• Example California Communities of Excellence in
  Tobacco Control
• OTIS Featurescontracts, progress reports,
  electronic approvals, cost reports
• Statewide reporting of Core IndicatorsETS,
  availability, countering pro-tobacco influences,
  assets, cessation ( s/w activity, costs)
• Reportscoalition activities, challenges/barriers,
  staffing, progress, evaluation plan, budgets/w
  evaluation comes from s/w surveys
• Many lessons learned

13
SNE/ERS Highlights FNS EARSFormation of an
Education Administrative Reporting System

• Objectives Develop standard state reports and
  uniform data elements paint picture of state
  activities
• Uses Inform management decisions, support policy
  initiatives document for legislative, budget and
  other requests support planning functions
• Evaluation niche Describe target groups, gaps,
  channels, resources
• Progress to date Public notice, appoint Working
  Group, review S/A reports, establish subcontracts
  for facilitator and field testing
• To do Draft report, synthesize comments, field
  test, get OMB clearance implement by FFY 05

14
SNE/ERS Highlights CSREES Logic
Modelwww.csrees-fsnep

• OverviewInputs?Outputs?Outcomes/Impact (short,
  medium, long-term)
• Core ElementsDietary quality, food security,
  food safety, shopping behavior and resource
  management indicators
• InputsFinancial, planning, materials, people
• OutputsActivities, participation
• Three levelsIndividual and households
  (individuals, agencies, policy makers)
  Communities and institutions Social structures,
  policies, and practices

15
SNE/ERS Highlights CDC Perspectivewww.cdc.gov

• CDC is asked by Congress about FSNE issues!
• Model of Behavioral Epidemiology Guidelines,
  recommendations, training materials,
  funding?policies, programs, service delivery,
  provision of benefits?relevant knowledge,
  attitudes, skills?health behaviors?health, social
  outcomes, well-being, quality of life
• Cycle Surveillance?Define problem?Establish
  priorities?Develop program plan and objectives?
  Implement?Measure change and strengthen program
• Logic Framework Modifiable determinants (food
  supply, environment, consumer demand)?Population
  food intake?Community health outcomes
• Available surveillance systems BRFSS, NHANES,
  YRBSS, PedNSS, PNSS, SHPPS

16
SNE/ERS Highlights Build on Social Sciences
ResearchExamples from the Southern Rural
Development Center, MSU

• Multiple obesity and poverty, food assistance
  studies
• Retail globalization and food access in the South
• Relationships of food insufficiency to disease
  risk and outcomes
• Contextual determinants of food security in
  Hispanic and African-American neighborhoods
  assessment of faith-based food assistance
  programs
• Charitable agency profiles and directors
  perceptions of needs and opportunities
• Studies on EBT, micro- and macro-level
  determinants of Food Stamp utilization

17
SNE/ERS Highlights Observations of an Evaluation
SpecialistEllen Taylor-Powell, PhD

• Top 10 Requirements
• Strong leadership
• Common language
• Engaged stakeholdersfunders, partners, staff,
  clients
• Uses must be showcased for learning and internal
  value
• Start with the what, then go to the how

18
SNE/ERS Highlights Observations of an Evaluation
SpecialistEllen Taylor-Powell, PhD

• Top 10 Requirements
• Create an evaluation cultureits everyones
  responsibility
• Build-in incentive and reward systems
• Streamlined data collection system
• Excellent training and TA
• Investmentresearch studies, time, analytical
  expertise, collaboration

19
Proceedings of ERS January Workshop and SNE/ERS
Workshop Now Available at http//www.csrees-fsnep
.org/confproc.cfm
20
Benchmarks Whos Doing What Next?A Work in
Progress

• USDAInteragency Working Group, web info, CDC
  participation?
• FNSEARS contracted report on qualitative
  profile of S/A FSNE
• CSREESContinued work on Community Nutrition
  Logic Model, state reports
• ERSDeveloping research project to improve
  outcome measures developing FSNE page on ERS
  website with interactive data features for use in
  FSNE planning and evaluation
• ASNNAEvaluation Committee, individual states
  share systems at Winter meeting, development of
  indicators in outer spheres?
• GAOReport scheduled for April 04
• Congress, state agencies and legislatures???
• SNEsocial marketing division and scientific
  programs