Investigating the Management of Systemic Change Related to Professional Development March 16, 2004

1
Investigating the Management of Systemic Change
Related to Professional Development March 16,
2004

• Tom Fiore
• Alison Tanner
• Westat
• Larry Magliocca
• Ohio State University

2
Overview of the Presentation

• Brief overview of the SIG Program
• Brief overview of the SIG Program Evaluations
  key issue study on systemic change
• Model for examining management of systemic change
• Important attributes of systemic change
  strategies
• Discussion of issues to consider related to our
  findings and to systemic change in general

3
SIG Program

• State Program Improvement Grant Program was
  authorized by the 1997 Amendments to IDEA as a
  joint effort of the federal government and states
  to improve results for children with disabilities
• OSEP awards competitive grants to states to
  implement SIG projects based on State Improvement
  Plans

4
SIG Program

• OSEP made the first awards in 1999, each for 5
  years
• To date, OSEP has made awards to 47 states and
  D.C.
• Supplements, also awarded on a competitive basis,
  have increased the size of the grants

5
SIG Program Evaluation

• In 2000, OSEP funded Westat to conduct a
  formative evaluation of the SIG Program
• The overarching goal for the five-year Evaluation
  of the State Improvement Grant Program is to help
  OSEP improve the program within the timeframe of
  the programs operation

6
SIG Program Evaluation

• Not evaluating individual SIG projects
• Recently the evaluation has focused intensively
  on systemic change, which has always been a
  defining feature of the SIG Program

7
SIG Program and Systemic Change

• In establishing the SIG Program, Congress called
  for applicants . . .
• To use funds in reforming and improving their
  systems for providing educational, early
  intervention, and transitional servicesto
  improve results for children with disabilities
  Sec. 652 (b)
• And to describe how grant funds will be used in
  undertaking the systemic-change activities Sec.
  653(c)(2) Sec.

8
SIG Program and Systemic Change

• OSEP made this expectation explicit in its
  requests for grant applications from states . . .
• In determining the significance of the proposed
  project, the Secretary considers the likelihood
  that the proposed project will result in system
  change or improvement CFDA 84.323A

9
SIG Program and Systemic Change

• OSEP also made the systemic change purpose clear
  in developing the competition for funding our
  evaluationthe evaluation was expected to . . .
• Provide information and recommendations regarding
  the extent to which the SIG program is meeting
  three fundamental goals, including to improve
  results for infants, toddlers, and children with
  disabilities as an outcome of systemic change.
  84.329A

10
SIG Program and Systemic Change

• Despite systemic change being a defining feature
  of the SIG Program, neither Congress nor OSEP
  explained what characterizes systemic change
• Defining systemic change was left to the states

11
SIG Program and Systemic Change

• Systemic change is a central focus of Westats
  evaluation
• The starting point for defining systemic change
  for the purposes of our evaluation came from the
  literature and from what states indicated in
  their proposals that they would be doing

12
SIG Program and Systemic Change

• First, words from Michael Fullan
• Systemic reform is perhaps the most overused and
  misunderstood concept in education today. It is
  used loosely to refer to almost any attempt that
  a given author considers to be major. In fact,
  systems thinking has a strong tradition in
  theory, albeit with a much less well-developed
  knowledge base in practice.

13
SIG Program and Systemic Change

• Our starting pointa working definition of
  systemic change
• Systemic change is both a process and an outcome
  . . .
• - A process where reformers use strategies
  consistent with systems theory to change targeted
  systems
• - An outcome where changes to relevant systems
  occur that (a) alter how organizations and
  individuals operate and relate to each other and
  (b) result in improved outcomes for a target group

14
Data Collection

• Various data collection activities across 3 years
  have addressed systemic change within the SIG
  Program
• SIG project directors survey in 2002
• SIG project evaluators survey in 2002
• Site visits to 1999- and 2000- and 2001-cohort
  states, most multiple times
• Data extraction

15
Data Collection

• Various data collection activities (cont.)
• Interviews with federal officials involved in
  developing and administering the SIG Program
• Limited observations of OSEP activities and SIG
  project activities in a few states
• Two-day focus group with leaders of active SIG
  projects
• Extensive review of theoretical and research
  literature on systemic change

16
Examining the Management of Systemic Change

• Investigation of systemic change in the SIG
  Program thus far has resulted in a grounded
  theory of how states are attempting to achieve
  systemic change

17
Examining the Management of Systemic Change

• The grounded theory focuses on how states are
  managing the systemic change projects
• Developed through an iterative process, lasting
  almost a year

18
Examining the Management of Systemic Change

• The grounded theory is depicted through . . .
• A model of how states are managing the systemic
  change process
• A set of the interactive attributes that appear
  to be key to the successful implementation of
  systemic strategies

19
Model for Examining the Management of Systemic
Change
Environmental Changes
Feedforward
F
K
Needs Assessment Functions
Vision and Purpose-Setting Functions
Implemen- tation Functions
Planning and Decision-Making Functions

• Evaluation Functions
• Embedded Inquiry
• Formal Evaluation

B
C
D
E
Outcome State
I. Improved System Function
III. Improved Student Outcomes
II. Scaling- Up of Practices
A
J
Unique Contextual Factors
Feedback 1
G
I
Worth
Feedback 2
Reflec- tion
H
20
Model for Examining the Management of Systemic
Change
Environmental Changes
Feedforward
F
K
Needs Assessment Functions
Vision and Purpose-Setting Functions
Implemen- tation Functions
Planning and Decision-Making Functions

• Evaluation Functions
• Embedded Inquiry
• Formal Evaluation

Outcome State
I. Improved System Function
III. Improved Student Outcomes
II. Scaling- Up of Practices
A
J
Unique Contextual Factors
Feedback 1
G
I
Worth
Feedback 2
Reflec- tion
H
21
Needs Assessment Functions

• Collecting needs data
• Engaging stakeholders in dialogue about needs
• Prioritizing needs
• Documenting and justifying needs selection

22
Model for Examining the Management of Systemic
Change
Needs Assessment Functions
Vision and Purpose-Setting Functions
Implemen- tation Functions
Planning and Decision-Making Functions

• Evaluation Functions
• Embedded Inquiry
• Formal Evaluation

23
Vision and Purpose-Setting Functions

• Establishing and updating a vision of a desired
  outcome state
• Establishing an overall direction and purpose
• Committing to shared responsibility and
  accountability
• Considering environmental issues and changes
• Maintaining focus

24
Model for Examining the Management of Systemic
Change
Planning and Decision-Making Functions
Needs Assessment Functions
Vision and Purpose-Setting Functions
Implemen- tation Functions

• Evaluation Functions
• Embedded Inquiry
• Formal Evaluation

25
Planning and Decision-Making Functions

• Allocating resources
• Creating policies, procedures, guidelines,
  regulations
• Ensuring policy and other alignments
• Designing implementation strategies
• Adjusting to environmental issues and changes
• Using feedback to monitor implementation
• Making course corrections

26
Model for Examining the Management of Systemic
Change
Needs Assessment Functions
Vision and Purpose-Setting Functions
Implemen- tation Functions
Planning and Decision-Making Functions

• Evaluation Functions
• Embedded Inquiry
• Formal Evaluation

27
Implementation Functions

• Instituting policies and regulations
• Executing strategies and activities
• Providing and administering stipends, awards, and
  subgrants
• Establishing model demonstration, pilot sites
• Providing guidance, technical assistance,
  training, materials, and other support to
  customers
• Disseminating information to customers
• Collecting data

28
Model for Examining the Management of Systemic
Change
Needs Assessment Functions
Vision and Purpose-Setting Functions
Implemen- tation Functions
Planning and Decision-Making Functions

• Evaluation Functions
• Embedded Inquiry
• Formal Evaluation

29
Evaluation FunctionsEmbedded and Formal

• Evaluation is an intrinsic part of the reform
  design
• Embedded Inquiry is the ongoing collection and
  use of data that is built into the reform
• Formal Evaluation is the structured collection
  and use of data that is designed to answer
  specific formative and summative questions

30
Evaluation Functions

• Establishing metrics for outputs and outcomes
• Establishing measurable expectations for
  management and implementation
• Designing evaluation plan
• Collecting data
• Analyzing and interpreting data
• Disseminating findings
• Determining recommendations based on findings

31
Model for Examining the Management of Systemic
Change
Needs Assessment Functions
Vision and Purpose-Setting Functions
Implemen- tation Functions
Planning and Decision-Making Functions

• Evaluation Functions
• Embedded Inquiry
• Formal Evaluation

Outcome State
II. Scaling- Up of Practices
I. Improved System Function
III. Improved Student Outcomes
32
Outcome State

• Collection of integrated outcomes that
  comprehensively define the condition being
  achievedthe result of the reform
• Provides benchmarks for progress

33
Outcome State

• Focus is still on the vision, plus the actual
  results being achieved
• Placed within the Planning and Decision-Making
  Functions
• The outcome state is the focal point of the
  entire reform

34
Outcome State

• For reforms like the SIG projects, successful
  systemic change produces a changed outcome state
  in three ways . . .
• Improved System Function
• Scaling-Up
• Improved Student Outcomes

35
Model for Examining the Management of Systemic
Change
Needs Assessment Functions
Vision and Purpose-Setting Functions
Implemen- tation Functions
Planning and Decision-Making Functions

• Evaluation Functions
• Embedded Inquiry
• Formal Evaluation

Outcome State
I. Improved System Function
III. Improved Student Outcomes
II. Scaling- Up of Practices
Unique Contextual Factors
36
Unique Contextual Factors

• Historical events (state or local) relevant to
  the needs
• Cultural factors relevant to the needs
• Political factors relevant to the needs

37
Model for Examining the Management of Systemic
Change
Environmental Changes
Needs Assessment Functions
Vision and Purpose-Setting Functions
Implemen- tation Functions
Planning and Decision-Making Functions

• Evaluation Functions
• Embedded Inquiry
• Formal Evaluation

Outcome State
I. Improved System Function
III. Improved Student Outcomes
II. Scaling- Up of Practices
Unique Contextual Factors
38
Environmental Changes

• Effects of . . .
• Changing federal and state policies
• Changes in demographics and other state or local
  contextual factors
• Innovations
• Other anticipated or unanticipated changes

39
Connections and Questions

• Explained the model by describing the five
  functions and important influences on those
  functions
• Now describing the connections that make it
  systemic, and the questions that make the model a
  potential evaluation tool

40
Model for Examining the Management of Systemic
Change
Environmental Changes
Needs Assessment Functions
Vision and Purpose-Setting Functions
Implemen- tation Functions
Planning and Decision-Making Functions

• Evaluation Functions
• Embedded Inquiry
• Formal Evaluation

B
D
C
E
Outcome State
I. Improved System Function
III. Improved Student Outcomes
II. Scaling- Up of Practices
A
Unique Contextual Factors
41
Arrows

• While the boxes indicate what reformers are
  doing, the arrows focus on how they are doing it
• Each arrow is defined by a broad question, which
  serves as an overarching evaluation question

42
Arrows A-E

• Arrow A sets the reform in its state context
• Arrows B-E represent the direct connections
  across the major functions

43
Arrows A-E

• How are historical, political, and other
  contextual issues considered in the
  identification of the needs and the selection of
  the stakeholders involved in determining the
  needs?

44
Arrows A-E

• How are needs translated into the vision of the
  desired outcome state?
• How are decisions made about goals and broad
  strategies that transform the vision into plans?

45
Arrows A-E

• How do plans and ongoing decisions lead to
  collaborations, actions, and activities that
  pursue the vision?
• How are the outputs and outcomes associated with
  decisions and activities identified, selected,
  and measured?

46
Model for Examining the Management of Systemic
Change
Environmental Changes
Feedforward
F
Needs Assessment Functions
Vision and Purpose-Setting Functions
Implemen- tation Functions
Planning and Decision-Making Functions

• Evaluation Functions
• Embedded Inquiry
• Formal Evaluation

B
C
D
E
Outcome State
I. Improved System Function
III. Improved Student Outcomes
II. Scaling- Up of Practices
A
Unique Contextual Factors
47
Feedforward

• Feedforward is associated with leadership
• Proclaiming and sustaining the vision of the
  outcome state and associated ideals
• Promoting the expectation for change
• Planning contingencies/scenarios
• Anticipating results
• Thinking and planning (developing schemes)
  regarding sustainability

48
Feedforward Arrow

• How is the vision promoted and sustained?

49
Model for Examining the Management of Systemic
Change
Environmental Changes
Feedforward
F
Needs Assessment Functions
Vision and Purpose-Setting Functions
Implemen- tation Functions
Planning and Decision-Making Functions

• Evaluation Functions
• Embedded Inquiry
• Formal Evaluation

B
C
D
E
Outcome State
I. Improved System Function
III. Improved Student Outcomes
II. Scaling- Up of Practices
A
Unique Contextual Factors
Feedback 1
G
50
Feedback 1 Arrow

• First feedback looprepresented by Arrow
  Gconnects the Evaluation back to the Planning
  and Decision-Making
• How is evaluative information used to adjust
  plans, activities, and management?

51
Model for Examining the Management of Systemic
Change
Environmental Changes
Feedforward
F
Needs Assessment Functions
Vision and Purpose-Setting Functions
Implemen- tation Functions
Planning and Decision-Making Functions

• Evaluation Functions
• Embedded Inquiry
• Formal Evaluation

B
C
D
E
Outcome State
I. Improved System Function
III. Improved Student Outcomes
II. Scaling- Up of Practices
A
Unique Contextual Factors
Feedback 1
G
Worth
Feedback 2
Reflec- tion
52
Feedback 2

• Second feedback loopusing data and other
  information to determine the overall value of
  results and to reflect on the performance of
  organizations and systems in achieving the results

53
Arrow H

• How is evaluative information used to determine
  the worth or value of actions and activities
  vis-à-vis the desired outcome state?

54
Model for Examining the Management of Systemic
Change
Environmental Changes
Feedforward
F
Needs Assessment Functions
Vision and Purpose-Setting Functions
Implemen- tation Functions
Planning and Decision-Making Functions

• Evaluation Functions
• Embedded Inquiry
• Formal Evaluation

B
C
D
E
Outcome State
I. Improved System Function
III. Improved Student Outcomes
II. Scaling- Up of Practices
A
Unique Contextual Factors
Feedback 1
G
Worth
Feedback 2
Reflec- tion
H
55
Worth

• Using data and other feedback to make decisions
  regarding the value of the activities and related
  outcomes

56
Arrow I

• How are determinations of the worth of actions
  and activities used to alter plans and management
  processes?

57
Model for Examining the Management of Systemic
Change
Environmental Changes
Feedforward
F
Needs Assessment Functions
Vision and Purpose-Setting Functions
Implemen- tation Functions
Planning and Decision-Making Functions

• Evaluation Functions
• Embedded Inquiry
• Formal Evaluation

B
C
D
E
Outcome State
I. Improved System Function
III. Improved Student Outcomes
II. Scaling- Up of Practices
A
Unique Contextual Factors
Feedback 1
G
I
Reflec-tion
Worth
Feedback 2
H
58
Reflection

• Using data, other feedback, and considerations of
  worth to make decisions regarding the value,
  purpose, or effectiveness of pertinent
  organizations and systems

59
Reflection

• Using data, other feedback, and considerations of
  worth to make decisions regarding the relevance
  of the overall purpose, vision, or direction

60
Arrow J

• How do reflections on the worth of actions and
  activities lead to modifications of the
  vision/purpose, and alterations to organizational
  structures and management processes?

61
Model for Examining the Management of Systemic
Change
Environmental Changes
Feedforward
F
Needs Assessment Functions
Vision and Purpose-Setting Functions
Implemen- tation Functions
Planning and Decision-Making Functions

• Evaluation Functions
• Embedded Inquiry
• Formal Evaluation

B
C
D
E
Outcome State
I. Improved System Function
III. Improved Student Outcomes
II. Scaling- Up of Practices
A
J
Unique Contextual Factors
Feedback 1
G
I
Reflec-tion
Worth
Feedback 2
H
62
Environmental Changes Arrow

• How are relevant, external environmental changes
  identified and incorporated?

63
Model for Examining the Management of Systemic
Change
Environmental Changes
Feedforward
F
K
Needs Assessment Functions
Vision and Purpose-Setting Functions
Implemen- tation Functions
Planning and Decision-Making Functions

• Evaluation Functions
• Embedded Inquiry
• Formal Evaluation

B
C
D
E
Outcome State
I. Improved System Function
III. Improved Student Outcomes
II. Scaling- Up of Practices
A
J
Unique Contextual Factors
Feedback 1
G
I
Worth
Feedback 2
Reflec- tion
H
64
Summary

• Functions alone dont constitute a systemic
  reform
• What makes a reform systemic is . . .
• The relationships between the functions
• The loops

65
(No Transcript)
66
Introduction

• Topic
• Preliminary finding of Westats evaluation of
  systemic change in the SIG projects

67
Background

• Understand state-level systemic change from the
  perspective of SIG project staff and stakeholders
• Apply the existing theory and research on
  systemic change and education reform

68
Background

• How (systemically) are SIG projects implemented
  and managed?
• Integration of management functions
• Use of feedback loops
• Systemic processes

69
Background

• Interactive attributes of systemic change
  strategies
• Preliminary finding
• Anticipated utility
• Open to feedback

70
Interactive Attributes of Systemic Change
Strategies

• Established Needs
• Stakeholder Involvement
• Systemic Intentionality
• Sustainability
• Scientific Knowledge
• Vision
• Resource Allocation

• Scale
• Consideration of Context
• Communication
• Distributive Leadership
• Collaboration
• Data-Based Decision-Making
• Reflective Restructuring

71
Interactive Attributes of Systemic Change
Strategies

• Established Needs
• Stakeholder Involvement
• Systemic Intentionality
• Sustainability
• Scientific Knowledge
• Vision
• Resource Allocation

• Scale
• Consideration of Context
• Communication
• Distributive Leadership
• Collaboration
• Data-Based Decision-Making
• Reflective Restructuring

72
Interactive Attributes of Systemic Change
Strategies

• Established Needs
• Stakeholder Involvement
• Systemic Intentionality
• Sustainability
• Scientific Knowledge
• Vision
• Resource Allocation

• Scale
• Consideration of Context
• Communication
• Distributive Leadership
• Collaboration
• Data-Based Decision-Making
• Reflective Restructuring

73
Interactive Attributes of Systemic Change
Strategies

• Established Needs
• Stakeholder Involvement
• Systemic Intentionality
• Sustainability
• Scientific Knowledge
• Vision
• Resource Allocation

• Scale
• Consideration of Context
• Communication
• Distributive Leadership
• Collaboration
• Data-Based Decision-Making
• Reflective Restructuring

74
Categories of Attributes

• Grounding Principles
• Design
• Operations
• Adjustments
• Desired Outcomes

75
Interactive Attributes of Systemic Change
Strategies
76
Interactive Attributes of Systemic Change
Strategies
77
Grounding Principles

• Established Needs
• Stakeholder Involvement
• Systemic Intentionality
• Sustainability
• Scientific Knowledge

78
Interactive Attributes of Systemic Change
Strategies
Established Needs
Scientific Knowledge
Systemic Intentionality
Stakeholder Involvement
Sustainability
Grounding Principles
79
Design

• Vision
• Resource Allocation
• Scale
• Consideration of Context

80
Interactive Attributes of Systemic Change
Strategies
81
Operations

• Communication
• Distributive Leadership
• Collaboration

82
Interactive Attributes of Systemic Change
Strategies
83
Adjustments

• Data-Based Decision-Making
• Reflective Restructuring

84
Interactive Attributes of Systemic Change
Strategies
85
Desired Outcomes

• Intermediate Outcomes
• Student Outcomes

86
Interactive Attributes of Systemic Change
Strategies
87
Contact Information

• Tom Fiore TomFiore_at_westat.com
• Alison Tanner AlisonTanner_at_westat.com
• Larry Magliocca magliocca.1_at_osu.edu
• SIG Program Evaluation Reports and
• presentation slides
• www.signetwork.org