A Training Companion to A Guide to Measuring Advocacy and Policy

1
A Training Companion toA Guide to Measuring
Advocacy and Policy

• KIDS COUNT
• Annie E. Casey Foundation

2
Key Questions
What are your questions about evaluation?
What kinds of evaluation does your organization
currently do?
What do you see as the benefits of evaluation for
advocacy and policy change efforts?
3
Two Specific Purposes of Guide

• Help grantmakers and advocates think about and
  talk about measurement of advocacy and policy
  change efforts
• Put forth a framework for naming outcomes
  associated with advocacy and policy work as well
  as directions for evaluation design

4
Intended Outcomes of Guide

• Greater acknowledgement of how evaluation fits in
  the world of advocacy and policy change efforts
• Greater acceptance of the role of evaluation for
  strategic learning
• Increased confidence about undertaking evaluation
  related to advocacy and policy change efforts

5
In a Nutshell

• The framework and directions provide a common
  way to identify and talk about outcomes and
  evaluation approaches.

6
What We Have Learned

• Evaluating Advocacy and Policy Change

7
Current Expectations

• Current expectations for evaluation of advocacy
  and policy change efforts are wide-ranging.
  Examples
• Performance measures amount of effort
• Outputs what work is accomplished
• Capacity ability to carry out advocacy
• Policy wins e.g., legislation passed
• Broad population indicators changes in lives

8
Two General Themes

• Lack of clear expectations
• Over-reaching expectations

9
In the Words of an Advocate

• Progressive funders constantly ask advocates and
  organizations to prove that our work results in
  policy change. Theyd like us to draw a straight
  line between our activities and the change we
  seek, year after year, and theyd like us to walk
  down that line quickly. The fact that social
  movements that feed truly large scale policy
  change dont work that way wouldnt be so
  unfortunate with progressive elites werent so
  attached to that idea, forcing the flow of
  resources into very narrow channels.

Rinku Sen, Leading La Marcha,
www.tompaine.com/articles (April 10, 2005)
10
Ten Key Lessons

• The major task of evaluation in the advocacy and
  policy arena is to define the middle section of
  advocacy and policy work.
• The time investment for this work is very
  challenging.
• Advocacy work involves cross-collaboration among
  many different partners (within organization and
  across organizations). Consider how and in what
  ways to engage partners in evaluation planning.

11
Ten Key Lessons

• The doing it all trap makes it hard to capture
  specific strategic outcomes that are the focus
  for evaluation, learning and accountability.
• Advocacy organizations will stress the importance
  of maintaining flexibility to change or modify
  outcomes to best reflect the developments in the
  contextual factors of as advocacy and policy
  change work.

12
Ten Key Lessons

• Public sharing of outcomes or evaluation data may
  be problematic.
• There is likely to be great variation in
  backgrounds among the people involved with
  advocacy and policy work, including legal,
  political, public relations, communication,
  business and other discipline outside traditional
  non-profit sector work.
• There is likely to be strong interest in data
  (e.g., community indicators), but not evaluation
  due to the elusiveness and transience of advocacy
  and policy outcomes.

13
Ten Key Lessons

• It will be important to equate asking useful
  questions with evaluation--in contrast to a
  strong reliance on victory stories or measuring
  the ultimate impacts on populations.
• Upper management involvement in the
  development of theory of change, defining of
  outcomes and planning for evaluation will be
  important to the ultimate success of this work.

14
In the Words of an Evaluator

• We should not be daunted by the methodological
  challenges. . . . We have to push ahead we have
  to try some new things. We have to put data
  collection strategies into the field even if they
  are imperfect, try them and work on their
  development.

Gary Henry, Ask the Experts, Evaluation
Exchange, April 13, 2006
15
In a Nutshell

• It is an important and exciting time to advance
  this evaluation field and work through these
  challenges.

16
Getting Started

• Evaluating Advocacy and Policy Change

17
Three Steps to Getting Started

• Start with a theory of change
• Identify outcomes and your outcome model
• Select a practical and strategic approach to
  measurement

18
Step One Start with a Theory of Change

• Evaluating Advocacy and Policy Change

19
Tip Express Your Theory of Change
Create an intentional map or model of how change
happens. Not only will you create a useful
concrete product, but you will also experience a
valuable collaborative process in developing the
product.

• TIP

20
What is a Theory of Change?

• Description of the belief system (e.g., evidence,
  assumptions, best practices, community
  experience) and research base for making positive
  change
• Conceptual model for achieving a collective
  vision

21
Benefits of a Theory of Change

• Develops specificity about high level strategies
  and outcomes
• Fosters common agreement about impact (i.e.,
  social change, policy change, advocacy)
• Results in a concrete product as well as a
  valuable collaborative process of development

22
Theory of Change Models

• Logic Model

• Outcome Map

• Clear graphic format in precise language that
  represents linkages between resources,
  activities, outputs, outcomes, and goals

• Visual diagram that depicts a set of strategies,
  outcomes and goals and the logical
  interconnections among them

23
Mothers Against Drunk Driving Theory of Change
Increased readiness of advocacy organizations to
advocate support for public policies that would
prevent drunk driving, alcohol-related accidents
and underage drinking
Increased public support for public policies that
would prevent drunk driving, alcohol-related
accidents and underage drinking
Increased public knowledge of the problems of
drunk driving, victims of alcohol-related
accidents and underage drinking
Increased public concern for, and decreased
tolerance of, drunk driving, alcohol-related
accidents, and underage drinking
Lives are saved
24
Opportunity Prosperity Campaign, Connecticut
Association for Human Services DRAFT Theory of
Change
25
Step Two Identify Outcomes and Your Outcome Model

• Evaluating Advocacy and Policy Change

26
Identifying Outcomes
There are many ways to frame advocacy and policy
change outcomes. We have organized the general
outcome areas into six buckets.
27
Advocacy and Policy ChangeOutcome Buckets

• Shift in Social Norms
• Strengthened Organizational Capacity
• Strengthened Alliances
• Strengthened Base of Support
• Improved Policies
• Changes in Impact

28
Outcome Bucket Shift in Social Norms

• The knowledge, attitudes, values and behaviors
  that compose the normative structure of culture
  and society

29
Shift in Social Norms

• Examples of Outcomes

• Examples of Strategies

• Changes in awareness
• Increased agreement on the definition of a
  problem
• Changes in beliefs
• Changes in attitudes
• Changes in values
• Changes in the salience of an issue
• Increased alignment of campaign goals with core
  societal values
• Changes in public behavior

• Framing issues
• Media campaign
• Message development
• Development of trusted messengers and champions

30
Outcome BucketStrengthened Organizational
Capacity

• The skill set, staffing and leadership,
  organizational structure and systems, finances
  and strategic planning among non-profit
  organizations and formal coalitions that plan and
  carry out advocacy and policy work.

31
Strengthened Organizational Capacity

• Examples of Outcomes

• Examples of Strategies

• Improved management of organizations
• Improved strategic abilities
• Improved capacity to communicate and promote
  messages
• Improved stability of organizations

• Leadership development
• Organizational capacity-building
• Communication skill-building
• Strategic planning

32
Outcome Bucket Strengthened Alliances

• The level of coordination, collaboration and
  mission alignment among community and system
  partners.

33
Strengthened Alliances

• Examples of Outcomes

• Examples of Strategies

• Increased number of partners supporting an issue
• Increased level of collaboration
• Improved alignment of partnership efforts
• Strategic alliances with important partnership
  efforts
• Increased ability of coalitions working towards
  policy change to identify the policy change
  process

• Partnership development
• Coalition development
• Cross-sector campaigns
• Joint campaigns
• Building alliances among unlikely allies

34
Outcome BucketStrengthened Base of Support

• The grassroots, leadership, and institutional
  support for particular policy changes.

35
Strengthened Base of Support

• Examples of Outcomes

• Examples of Strategies

• Increased public involvement in an issue
• Increased level of actions taken by champions of
  an issue
• Changes in voting behavior
• Increased breadth of partners supporting an issue
• Increased media coverage
• Increased awareness of campaign
• Increased visibility of the campaign message
• Changes in public will

• Community organizing
• Media campaigns
• Outreach
• Public/grassroots engagement campaign
• Voter registration campaign
• Coalition development
• Development of trusted messengers and champions
• Policy analysis and debate
• Policy impact statements

36
Outcome Bucket Improved Policies

• The stages of policy change in the public policy
  arena. These stages include policy development,
  policy proposal, demonstration of support,
  adoption, funding, and implementation.

37
Improved Policies

• Examples of Outcomes

• Examples of Strategies

• Policy development
• Policy adoption
• Policy implementation
• Policy enforcement

• Scientific research
• Development of white papers
• Development of policy proposals
• Pilots/demonstration programs
• Educational briefings of legislators
• Watchdog function

38
Outcome Bucket Changes in Impact

• The ultimate changes in social and physical
  lives and conditions that motivate policy change
  efforts. Changes in impact are long-term
  outcomes and goals.

39
Changes in Impact

• Examples of Outcomes

• Examples of Strategies

• Improved social and physical conditions (e.g.,
  poverty, habitat diversity, health, equality,
  democracy)

• Combination of direct service and
  systems-changing initiatives

40
Example Activities and Outcome Categories
41
Select a Practical and Strategic Approachto
Measurement

• Evaluating Advocacy and Policy Change

42
Choosing Your Approach

• There are many different forms and directions
  for evaluation.
• Evaluation can be . . .
• Explicit and straightforward
• Expansive, broad-based and elaborate
• Real-time and immediate
• Retrospective
• Knowledge-building or field-building
• Scientific
• . . . and more!

43
Directions of Evaluation

• Identification and measurement of core outcome
  areas related to social change or policy change
• Evaluation of strategic progress
• Identification and measurement of short-term
  incremental objectives
• Assessment of the capacity of the advocacy and
  policy organization
• Case study documentation of process and impacts

Process
Capacity
Impacts
44
Data Collection Considerations

• What will be the level of rigor?
• From whom will data be collected?
• When will data be collected?
• What type of questions will the data address?
• What data might be already available?
• What kind of capacity exists to conduct data
  collection?
• What are the primary audiences for the data?

45
Connecting Evaluation Questionswith Policy
Change Questions
46
Connecting Evaluation Questionswith Policy
Change Questions
47
Connecting Evaluation Questionswith Policy
Change Questions
48
Connecting Evaluation Questionswith Policy
Change Questions
49
Resources
50
A Guide to Measuring Advocacy and Policy

• Puts forth a framework for naming outcomes
  associated with advocacy and policy
• Offers some key directions for evaluation design
  that include a broad range of methodologies,
  intensities, timeframes and purposes

51
A Handbook of Data Collection Tools

• Companion to A Guide to Measuring Advocacy and
  Policy
• Provides examples of practical tools and
  processes for collecting data for difficult
  advocacy outcome categories
• Gives some tips on ways to collect, analyze and
  use data from specific tools

52
Advocacy Progress Planner - Composite Logic Model

• Online tool for developing policy/ advocacy logic
  model based on a composite logic model

Planning.ContinuousProgress.org
53
KIDS COUNT Evaluation Toolkit

• Offers tools and instruments developed by
  Innonet for the KIDS COUNT Network (e.g., User
  Feedback Tool, Media Tracking Tool)
• Includes tools and related data entry
  spreadsheets

54
Alliance for Justice Tools

• Advocacy Evaluation Tool
• Advocacy Capacity Assessment Tool

55
The Challenge of Assessing Policy and Advocacy
Activities, Parts I and II

• Developed following the National Convening on
  Advocacy Evaluation
• Funded by and prepared for The California
  Endowment by Blueprint Research Design

http//www.calendow.org/uploadedFiles/Publications
/Evaluation/challenge_assessing_policy_advocacy.pd
f http//www.calendow.org/uploadedFiles/challenge_
assessing_policy_advocacy2.pdf
56
Innonet.org Advocacy Evaluation

• Online resources
• Stories
• Projects

57
Innonet.org eNewsletter
58
Spring 2007 Evaluation Exchange

• Effort to help build the field of advocacy and
  policy evaluation
• Discusses whats different about evaluating
  advocacy and policy change efforts

59
Wrapping Up
Questions
Comments
Ah-has