Planning State Change and Improvement Initiatives

1
Planning State Change and Improvement Initiatives

• Mini-Training Institute
• March 29, 2007
• Prepared by
• Judith A. Alamprese, Abt Associates Inc.
• judy_alamprese_at_abtassoc.com

2
Institute Outcomes

• Learn about
• Why state change or program improvement may be
  needed
• How to determine what should be changed using the
  AIDDE process
• Key steps in planning change or an improvement
  initiative

3
Institute Process

• Review current practices in collecting,
  analyzing, and using data
• Learn about AIDDE process for program analysis
  and improvement
• Apply the AIDDE model to a state case study
• Learn key elements of an improvement initiative

4
Group Activity

• What are two statistics (e.g., attendance
  hours, enrollment rates) about your programs that
  you analyze frequently or would like to analyze?
• What are two questions about the operation of
  your local programs that you would like to
  investigate further?

5
States Role in Improvement and Management
INFLUENCING FACTORS
STATE ADULT EDUCATION ACTIVITIES INVOLVING THE
USE OF DATA
INTENDED OUTCOMES
6
Program Improvement Initiative Exercise 1

• What internal and external priorities have
  prompted you to examine data or to make changes
  in your system?
• List two types of data/information that you
  collect.
• List the ways in which you have analyzed or would
  like to analyze these data.
• What actions have you taken or would you like to
  take based on these data/information?

7
System of ABE Program Operations
Community Environment
Institutional Capacity
Program Management
Learner Recruitment Orientation Intake Placement
Learner Personal Development
Program Intensity
Program Improvement/ Professional Development
Instruction Post-Assessment
Pre-Assessment
Learner Skill Development
Support Services
Interagency Collaboration
Learner Transition
8
Program Improvement Initiative What Data Tell Us

• State Adult Education Data
• Primary Data Sources
• Annual, semi-annual, or other scheduled reports
  on learners served (used in NRS reporting)
• Annual reports on program goals met, services
  offered, other aspects of program operations,
  financial data
• Data matching of learners (employment,
  participation in post-secondary education)
• Follow-up surveys of learners
• Program applications
• Program/staff surveys that state conducts or
  commissions
• Program monitoring reports
• Program evaluations (survey, case study,
  observation)

9
Program Improvement Initiative What Data Tell Us

• State Adult Education Data
• Characteristics of Learners Enrolled
• Demographic/background characteristics
• Labor force status
• Educational gains
• Education and Employment Outcomes
• Participation
• Employment outcomes
• Community participation
• Family outcomes
• Learners perceptions of instructional services

10
Program Improvement Initiative What Data Tell Us

• State Adult Education Data
• Characteristics of Instructional Staff/Practices
• Background of instructional staff
• Staffs employment and training experience
• Characteristics of instructional approach
• Quality of Program Operations
• Recruitment
• Learner intake
• Learner placement
• Structure and hours of instruction

11
Program Improvement Initiative What Data Tell Us

• State Adult Education Monitoring/Annual Report
  Data
• Quality of Program Operations
• Assessment instruments and procedures
• Support services
• Learner transition
• Fiscal management
• Program management
• Interagency collaboration
• Data management

12
Program Improvement Initiative What Data Tell Us

• Program Data
• Provide information about each programs learners
  in a state
• Enable the state to compare programs concerning
• The demographic characteristics of learners
• The amount of contact hours that learners receive
• Whether or not learners make educational progress
• Whether the type of employment status of staff is
  related to whether learners make progress

13
Program Improvement Initiative What Data Tell Us

• Data Analysis
• Limited analyses are possible with data
  aggregated across all programs
• Data disaggregated by program provide opportunity
  for comparative analyses
• Student-level data system enables state to
  conduct more varied analyses

14
Program Improvement Initiative What Data Tell Us

• Related State and Regional Data
• Can be used as benchmark for state adult
  education data and to promote program
• May require establishing collaborative
  relationship with state demographer/data office,
  state labor department, etc.
• Examples of comparisons
• Extent to which state adult education program is
  serving target population of adult learners
• Identification of areas of state that might
  target services to dislocated workers, other
  populations
• Comparisons of employment and postsecondary
  outcomes

15
AIDDE Process for State Program Improvement
Initiative
Document Activities and Outcomes
Analyze Learner Data, Program Services,
Policies, and Pressures
Evaluate Implementation of Activities and
Outcomes
Develop Plan for Program Improvement Initiative
Conduct Workshops and Provide Assistance
Identify Areas for Improvement Initiative and
Approaches
16
Steps in the AIDDE Process

• Analyze Learners
• Background characteristics
• Performance data

Understanding of types of learners who access
services, their goals, and their progress
PROVIDES
Insights into strengths and limitations of
services, areas in need of enhancement
PROVIDES
Analyze Program Services

• Analyze Policies
• Federally directed policies
• State-initiated policies

Information for understanding quality of
programs services and performance issues
PROVIDES
Understanding of external and internal priorities
that may influence decisions about direction of
program improvement initiative
PROVIDES
Analyze Pressures
17
Relationship Between State Policy and ABE Program
Components
State Policy
Recruitment
Orientation
Intake/ Placement
Assessment
Instruction
ABE Program Components
Support Services
Transition
Judith Alamprese, Abt Associates 2005
18
Steps in the AIDDE Process
Identify Areas for Improvement Initiative
Focus for staff to use in planning initiative
PROVIDES

• Identify Approaches
• Conduct small pilot initiative before involving
  all programs
• Develop a phased schedule of training with
  subsets of programs

Strategy for developing support for initiative
and testing process before going statewide
PROVIDES
19
Steps in the AIDDE Process
Develop Plan for Improvement Initiative
Steps in the process for developing and
implementing an improvement initiative
PROVIDES
Determine Focus and Approach of Initiative
PROVIDES
Strategy for focusing on one improvement area
across programs or giving programs choice in what
they address
Identify Resources, Time-line and Expertise
PROVIDES
Information to use in planning time, funding, and
staff
A way of monitoring improvement activities,
addressing concerns
Develop Process for Ongoing Communication
PROVIDES
20
Steps in the AIDDE Process
Document Activities Related to Training and
Provision of Assistance
Data about the activities that were carried out
and the factors affecting these activities
PROVIDES
Identify Personnel, Materials, Resources Used
Information for planning funding, staff, and
resources that are needed to support an initiative
PROVIDES
Identify Barriers and Factors Facilitating
Outcomes
Information for solving problems related to the
implementation of the initiative
PROVIDES
21
Steps in the AIDDE Process
Evaluate Activities Implemented and Not
Implemented
PROVIDES
Information about aspects of initiative that were
difficult to implement
Perspective from staff about their perceptions of
activities effectiveness
Assess What Worked and What Didnt Work
PROVIDES
Identify Missing Components of Initiative
Ideas about additional activities that may need
to be undertaken to achieve outcomes
PROVIDES
Provides information for determining whether to
continue or adapt activities
Describe and Assess Results from Activities
PROVIDES
22
Elements of a Successful Program Improvement
Initiative

• Social-Emotional Elements
• Program staff feel an urgency to change their
  practice
• The vision for change is shared by multiple
  stakeholders
• Key opinion leaders promote the change/provide
  leadership
• Morale support and reinforcements are ongoing
• Constructive feedback is used to keep the
  initiative on track and identify possible barriers

• Policy Elements
• Initiative is linked to state adult education
  policies
• Guidance or policies are developed to support new
  practices in initiative
• Top-level support reinforces initiative
• Initiative is designed with pilot and revision
  phases

• Programmatic Elements
• Vision for initiative is clear and makes
  programmatic sense
• Steps involved in change process are delineated
• Resources are sufficient
• Time planned is adequate
• Qualified staff are available to carry out
  activities
• Training and technical assistance are provided
• Data are collected for feedback, refinement, and
  accountability

23
Characteristics of Program Improvement
Initiatives

• Program improvement initiatives can range in
  size, complexity, and timeline
• Size and complexity depend on the type of
  improvement that is to be undertaken, the number
  of programs that are to be involved, and the
  resources that are available
• The timeline must provide for multiple phases of
  activities planning, design, development,
  training, pilot implementation, ongoing technical
  assistance, revision, and additional
  implementation phases

24
Key Steps in a Program Improvement Initiative

• Identify the issue(s)/problem(s) to be addressed
  based on the analysis of data, practices,
  policies, and other factors
• Specify the expected outcomes
• Identify focus and magnitude of improvement
  activities
• Identify the participants, timeline, staff, and
  resources
• Specify training, technical assistance, and data
  collection activities
• Develop mechanisms for providing support and
  feedback to participants
• Develop procedures for analyzing data and
  reporting results

25
Assumptions Guiding AIDDE Improvement Process

• All levels in the systemstate policymakers,
  program directors, instructional staffare
  critical to program improvement
• Leadership is a shared function (distributed
  leadership)
• A heuristic model of an ABE program is a useful
  guide to understanding program functioning and
  improvement
• While there are few experimental results about
  the effective operation of ABE programs, existing
  research and data analysis strategies can be used
  in improving programs

26
States Use of AIDDE Process Improvement Areas

• Alaska, Idaho, Montana, and Oregon Programs
• Recruitment 6
• Retention 9
• Orientation 9
• Intake 11
• Assessment 19
• Instruction 10
• Learner Transition 6
• Program Management 3
• Program Improvement 9

27
Use of AIDDE Process Illustrative Results

• Recruitment/Retention
• Fewer students are falling through the cracks
• Reached new population of students
• 37 increase in students between 03 and 04
• Orientation
• Written processes have helped to standardize
  orientation
• Scheduling orientation session has helped program
  efficiency
• Assessment
• More students being post-tested
• Greater consistency in assessment procedures

28
Use of AIDDE Process Illustrative Results

• Instruction
• Improved classroom dynamics
• Developed class-based literacy instruction
• Learner Transition
• More follow-up for students pursuing GED
• Strengthened relationship with One-stop partners
  
• Communication
• More staff are feeling engaged and connected to
  their colleagues
• Greater communication between staff and students