Data and Analysis: Knowing Your Agency and Improving Your Results

1
Data and Analysis Knowing Your Agency and
Improving Your Results

• Phil Basso Robin OBrien
• June 4, 2007

2
Introductions

• Who We Are
• American Public Human Services Association
• Leadership and Practice Development Unit
• Organizational Effectiveness Products and
  Consulting

3
Session Objective

• Provide models, tools and examples to help you
  maximize your data analysis and functional
  capacity
• in order to
• Continuously improve your agencys performance
  and performance capacity.

4
Session Overview

• Part I Knowing Your Agency
• OE a Systemic and Systematic Approach to
  Continuous Improvement
• Markers of Organizational Effectiveness and
  Strategic Management Tools
• Part II Improving Your Results
• Assessing Your Current Data Analysis and
  Functional Capacity
• Planning Improvements
• Implementing and Monitoring Improvements

5
What is Organizational Effectiveness?

• OE is a systematic and systemic approach to
  continuously improving an organizations
  performance, performance capacity, and client
  outcomes.

6
Systemic Effectiveness OE Drivers

• What drives customer outcomes???
• Well designed and delivered customer services,
  enabled by-
• Finance and budgeting, workforce, and functional
  capacity, supported by-
• A foundation of data and analysis, values, and
  trust, all pulled together by-
• Assessing and planning improvements to the
  organizational system as a whole.

7
Systemic Effectiveness Levels of Work

• What work does the agency need to do???
• Executives accomplish strategy work by
  articulating what the agency aims to do, how well
  it does, and what the priorities are for
  improving it.
• Senior managers accomplish structure, culture and
  leadership work by setting boundaries and
  defining roles, monitoring performance, and
  modeling values.
• Middle-managers accomplish key processes work by
  determining specific processes and procedures
  that translate strategy into guidance for
  day-to-day work.
• Staff and supervisors accomplish operations work
  by using both key processes and their discretion
  to drive daily performance.

8
The Organizational System
S t r a t e g y
Performance
Inputs People Materials Equipment Finances
Performance Capacity
Outcomes Health Safety Permanency Self-Sufficienc
y
Outputs Services Products Relationships
Data Analysis Trust Values Budgeting Fiscal
Capacity Workforce Capacity Functional
Capacity Service Delivery
Environment
9
Systematic Effectiveness DAPIM

• In order to improve or innovate something, one
  must
• Define what that thing is in operational terms
• Assess its current and desired state
• Plan both rapid and long-term improvements
• Implement those plans in detail, and
• Monitor plan progress and impact for ongoing
  adjustment.

10
Systematic Effectiveness DAPIM
Define
Monitor
Assess
Performance Capacity
Implement
Plan
11
1. Define
12
The Organizational System
S t r a t e g y
Performance
Inputs People Materials Equipment Finances
Performance Capacity
Outcomes Health Safety Permanency Self-Sufficienc
y
Outputs Services Products Relationships
Data Analysis Trust Values Budgeting Fiscal
Capacity Workforce Capacity Functional
Capacity Service Delivery
Environment
13
Organizational System DrilldownElements of a
Strategic Playbook

• Vision and Mission
• Environmental Scan
• Principles, Values and Competencies
• Logic Model and Goals
• Performance Dashboard
• Organizational Scan
• Objectives and Key Initiatives
• Organizational Structure
• Performance Management System
• Business Case

14
Logic ModelsFour Key Benefits

• Helps define specific outcomes, goals,
  initiatives, outputs and measures of progress.
• Clarifies cause-and effect relationships and
  hypotheses for initial assessment and ongoing
  evaluation.
• Establishes the basis for specific agency
  strategies, change plans and organizational
  development.
• Establishes business cases, proofs, and a solid
  basis to communicate, make offers and requests,
  and partner.

15
Logic ModelsFour Key Challenges

• Comprehensive vs. Limited Paths
• Problem-Solving vs. Defending
• Chains-of-Evidence vs. System Flows
• Continuous Learning Studies vs. Static Pictures

16
Performance DashboardMeasuring Outputs and
Outcomes

• Define targets with your staff, customers and
  partners.
• Use your logic model to build measures around
  cause-and-effect linkages (e.g., logic models).
• Establish agency and operating unit dashboards
  through a strategic management program (e.g., a
  Balanced Scorecard).
• Differentiate measures between staff capacity,
  financial/efficiency, output, customer
  satisfaction, and customer outcomes.
• Longitudinal and cohort group data are essential
  ingredients to relevant analysis.

17
Strategic Performance Management SystemOverview

• Translates strategy into linked individual, unit,
  and departmental goals, deliverables and measures
• Organizes into a manageable framework the many
  strategic and change initiatives going on in a
  large, complex organization
• Blueprints how these initiatives are to be managed

18
Strategic Performance Management SystemBenefits

• Keeps leaders, staff and external stakeholders
  focused on organizational capacity and outcomes
  (versus outputs and compliance only)
• Enables alignment with strategy
• Resource allocation
• Goal setting and performance evaluation
• Day-to-day work
• Facilitates measurement and communication of
  Return on Investment (ROI)

19
Three Real-World Examples

• University of Pittsburgh Child Welfare Training
  Center
• Logic Model
• Hennepin County Human Services Agency
• Strategic Performance Management System
• Ventura County Human Services Agency
• Performance Dashboard

20
Group Discussion

• When you review the examples
• Any Ahas?
• How do these examples compare with others with
  which you have experience?
• How do you feel when comparing this approach to
  those you currently use Anxious? Validated?
  Other?

21
Preview of Part II

• Using markers of effectiveness to assess current
  agency capacity and identify key findings
• Driving rapid change concurrent with planning for
  systemic change
• Bridging the gap to action identifying
  priorities, root causes, and general remedies
• Preparing for action developing change plans
  that drive lasting, systemic improvements
• Driving and sustaining change implementing and
  monitoring change initiatives

22
Quick Review of Part I
23
Systematic Effectiveness DAPIM
Define
Monitor
Assess
Performance Capacity
Implement
Plan
24
The Organizational System
S t r a t e g y
Performance
Inputs People Materials Equipment Finances
Performance Capacity
Outcomes Health Safety Permanency Self-Sufficienc
y
Outputs Services Products Relationships
Data Analysis Trust Values Budgeting Fiscal
Capacity Workforce Capacity Functional
Capacity Service Delivery
Environment
25
Organizational System Drilldown Data Analysis
and Functional Capacity
26
Organizational System Drilldown Functional
Capacity
Strategic Influence
Leadership Influence
Functional Credibility Capacity
Tactical Expertise
Service Delivery
27
2. Assess
28
Defining Success for Your Agency

• To identify your agencys strengths and
  needs, you have to establish markers of
  effectiveness. Sources of effectiveness markers
  include
• APHSA-developed markers
• Accreditation Standards
• Benchmarking / Literature Reviews
• Markers Developed In-House
• Other????

29
Group Discussion

• When you review the APHSA markers of
  effectiveness
• Any Ahas?
• Do any markers jump out at you because of
  particular experiences youve had?
• How do you feel when considering these markers in
  light of your agencys current activities and
  capacity Anxious? Validated? Other?

30
Getting to FindingsGAP
Process
Analyze
Gather
31
Gathering DataData Sources

• Audit records
• Surveys, interviews and focus groups
• Staff, customers, partners.
• Structured observation
• Standard and ad-hoc reports
• Activity logs
• Data and analysis from community partners
• Comparative data and analysis
• Others???

32
Analyzing Data

• Review hard copy materials first
• Generate tentative hypotheses
• Identify people to interview (e.g., consider the
  sample size and mix you need to test hypotheses
  with confidence)
• Define structured interview questions
• Open ended
• Designed to not lead the witness
• Conduct interviews and analyze notes
• Refine tentative hypotheses into draft findings
• Scan markers of effectiveness for
  comprehensiveness

33
Processing DataThe Bridge to Planning
Gather and Analyze Data, then Process into

• Findings Strengths and Needs
• Priority Needs
• Root Causes
• General Remedies

• Rapid Changes, and Begin to Implement

34
Identifying Priority Needs
35
Identifying Root CausesTwo Complementary
Approaches

• Keep asking What isnt working well, and
  why? for a priority need until you discover
  something very tangible and actionablea general
  remedy
• or
• test a priority need against a set of
  categories of root causes and remedies until one
  or more fit.

36
Identifying Root CausesGeneral Categories

• More Relational
• Culture and Values
• Politics and Power
• Communication
• Teamwork and Collaboration
• Community Partnerships
• Daily Behaviors and Motivations
• Staff Capacity
• Time Management
• Skill Sets (technical/general)

• More Structural
• Organizational Structure
• Staffing Levels
• Levels and Roles
• Goals, Standards and Measures
• Decision-Making
• Policies and Procedures
• Processes and Methods
• Internal Programs, Services, and Tools

37
Identifying General Remedies

• There are three general types of remedies
• Recommendations
• Remedies are not in your control
• Decisions / Commitments
• Remedies are in your control and dont require
  development of new tools / processes
• Team Activities
• Remedies are in your control and require
  development of new tools / processes

38
Driving Rapid Change

• Pursue rapid change in areas that
• Enhance key initiatives already underway.
• Address needs that are obvious, big, and
  essential to client service delivery and
  relationship stability.
• Build the involvement and the confidence of
  clients and staff- quick, visible wins.
• Stretch current resources to an untried level,
  yet
• Eliminate or curtail lower-priority capacity
  eaters.
• Enable deeper future assessments and planning.
• Are symbolically powerful.

39
3. Plan
40
Purpose of a Change Plan

• Commit explicitly to improving client outcomes
  and services over time
• Build agency credibility with stakeholders and
  staff
• Get everyone on the same page about
  organizational improvement work to be done, how
  it will be rolled out over time, how different
  initiatives are connected, and why they matter
• Reinforce a culture of accountability,
  data-driven assessment, follow-through and
  ongoing adjustment

41
Group Discussion

• When you review the Change Planning Template
• Any Ahas?
• Do any elements of the change plan resonate
  particularly strongly with your experience?
• How do you feel when comparing this approach to
  those you currently use Anxious? Validated?
  Other?

42
4. Implement 5. Monitor
43
Implementation Steps

• Prepare for implementation for each major
  initiative in the change plan
• Charter action teams
• Create detailed action plans
• Develop ongoing evaluation and monitoring tools
  and processes
• Make public commitments
• Execute detailed action plans

44
MonitoringThree Techniques

• Implementation Reviews
• Measure accomplishments vs. plan milestones and
  commitments
• Impact Reviews
• Measure actual vs. expected impact on
  organizational capacity and customer outcomes
• Lessons Learned Reviews
• Address new and emerging questions and findings
  drive further innovations

45
Systematic Effectiveness DAPIM
Define
Monitor
Assess
Performance Capacity
Implement
Plan
46
APHSA Resources Available To You

• Indirect Resources
• Conference presentations, webinars, etc.
• Direct Products and Services
• Learning-By-Doing Institutes
• Customized Consulting
• Self-Directed Products (Coming Soon)
• For OE Alumni

47
For More Information

• feel free to contact us
• Phil Basso, Program Director, OE Products and
  Consulting
• pbasso_at_aphsa.org / 202-682-0100 ext. 227
• Robin OBrien, Program Manager, OE Products and
  Consulting
• robrien_at_aphsa.org / 202-682-0100 ext. 246