Lessons Learned from a Synergistic Program Evaluation Model Partnership Between a State Agency and University Program

1
Lessons Learned from a Synergistic Program
Evaluation Model Partnership Between a State
Agency and University Program

• Michael J. Leahy
• Michigan State University
• Program Evaluation Summit
• Salt Lake City, Utah
• September 25, 2008

2
Agenda

• Brief overview of MSU and MRS.
• Typical collaborations between university
  programs and State Agencies.
• Problems, issues and needs that led to the
  development of Project Excellence.
• Benefits of funded partnership to both parties.
• Project Excellence - scope of work and how it
  works.
• Lessons learned and success factors for PE in the
  public rehabilitation program.

3
Project ExcellenceA Program Evaluation
Partnership
Michigan State University
Project Excellence
Michigan Department of Labor Economic
Growth-Rehabilitation Services
4

Michigan
State University Office
of Rehabilitation and Disability Studies Program
began in 1956 Masters degree program in
Rehabilitation Counseling CORE
Accredited Doctoral degree program in
Rehabilitation Counselor Education REACH Program
for professional development Ranked the 1
program in the country (tied with University of
Wisconsin-Madison) by US News and World Report
Rankings of graduate programs in rehabilitation
counseling Long-standing, mutually beneficial
relationship with the public rehabilitation
program in Michigan
5
Department of Labor and Economic Growth -
Rehabilitation Services (MRS)

• 1.3 million people with disabilities in the
  Michigan - 27,000 citizens with disabilities
  served at any given time by MRS counselors.
• Over 7,600 customers placed in jobs in 2007.
• Core principles - customer-centered, equity,
  advocacy, collaboration, accountability, and
  continuous improvement
• 275 Rehabilitation Counselors.
• Heavy urban and isolated rural areas.
• History of innovation in service delivery
• Highest unemployment rate in the nation at 8.9
• High turnover -- early retirements of leadership

6

• My Initial Background in PE and
  Programmatic Research
• Crossroads Rehabilitation Center in Indianapolis
  Indiana
• CARF Section 9 Standards on Program Evaluation
  (mid 1970s)
• Trained in Program Evaluation through national
  CARF initiative
• Walker Model (Dr. Robert Walker)
• Became a trainer for Region V RRCEPS in program
  evaluation (late 1970s)
• Worked in this capacity for a number of years
  with community-based organizations throughout the
  region

7

• Typical Collaborations between a State Agency and
  a University Program
• Membership on Advisory Councils
• Input into graduate training programs
• Serving as adjunct instructors and guest
  lecturers
• Practicum and Internship sites for students
• Professional development for agency staff
• One substantive area that is not typically an
  integral aspect of the partnership
• is research and program evaluation
• Given the level of accountability for outcomes,
  the complexity of the process,
• and the limited resources of most state agencies
  to effectively devote time,
• energy and and expertise to program evaluation,
  this appears to be an ideal
• place for a university program to further
  contribute to the partnership.

8
Problems Viewed as
Opportunities MRS was experiencing a lack of
stability in the program evaluation area -
frequent turnover and difficulty finding and
retaining qualified personnel were persistent
problems. Agency very active in implementing a
series of new strategic directions and programs
and were unable to adequately measure the impact
of these initiatives and use data for continuous
improvement in performance. To have quality
programmatic research with timely information and
consistently evaluate programs over multiple
years became a greater need and objective of the
agency. Key factor - - State Director and
Executive staff value of PE.
9

• Benefits of the
  Funded Partnership
• MSU
• Opportunity to provide a rich, real world, field
  based environment
• to enhance the training of doctoral students in
  research and program evaluation.
• Provided a unique revenue stream to fund doctoral
  students as Research
• Assistants.
• MRS
• Stabilize and greatly expand its programmatic
  research and program evaluation
• capacity to meet routine regulatory reporting
  requirements.
• Use program evaluation for the first time, as a
  major component of its continuous
• improvement strategy for the agency.

10
Project ExcellenceA Program Evaluation
Partnership

• Began in 2001as a three-year project designed to
  enhance MRS program evaluation, outcome
  measurement, and reporting processes. After seven
  years in operation, will enter into a new grant
  in October for another five years (2008-2013).
• Administered by MSU Rehabilitation Program
  faculty, with two full-time doctoral level
  research associates, and six research assistants
  (doctoral students).
• Collaborative and consultative approach to
  working with MRS administrators counselors
  technical, training, and support staff and
  customers.

11
Project ExcellenceA Program Evaluation
Partnership

• Project Goal
• Project Excellence will further develop and
  improve the application of qualitative and
  quantitative data analysis with respect to
  Michigan Rehabilitation Services culture and
  programs while prioritizing regulatory
  responsibilities and program excellence values.

12
Project Excellence Scope of Work
13
How the Individual Projects are
Selected Each Year Each year the State Agency
calls for ideas and concerns that PE could
address from all levels of the
organization. These ideas are then reviewed with
the Execute staff to identify priority
areas. Once these potential projects are
prioritized the Agency and the PE staff meet to
review and make decisions about the scope of work
for the upcoming year. Each year, over the past
seven years, PE has had between 10-14
individual projects specified within the scope of
work.
14
Examples of Projects

• Annual review the RSA 911 Data
• Data Integrity Study
• Customer Satisfaction
• Services to Minority Populations
• Business Services
• Re-organization Impact Study
• MI Transition in the 21st Century
• Effectiveness of MRS sponsored CRO Grants
• Michigan Career and Technical Institute
• Orientation Process Evaluation
• Vendors Qualifications and Standards
• MI Comprehensive Needs Assessment
• Best Practices related to Successful Outcome

15
Project Excellence Flow of Operations
Implementation Team Senior Manager Input District
Manager Input Implementation Team
Review Training Dissemination Evaluation
Special Projects

• Ongoing
• Projects

Work Groups SMP Transition Youth Adjudicated
Youth MI HI Connections Customer
Satisfaction Business Services MCTI Grants
16
Lessons Learned - Success Factors

• Trust
• Commitment at the top of the organization a must
• Long history of collaborative efforts
• Learning organization with common values
• Creative leadership and management styles to
  address implementation of findings from PE -
  continuous improvement.
• Supportive Michigan Rehabilitation Council
• Enjoyment of projects, collaboration, and
  partnership
• Common goal to improve service delivery and
  enhance employment outcomes
• Common commitment to provide a variety of
  research opportunities to doctoral students

17
Doctoral Student Experiences

• What was gained
• Positive influences on development as
  researchers
• and leaders
• Approaches to team work on researchfrom
  concept
• to product
• Methods for increasing research skills via
  access to
• relevant databeyond 911 data.

18
Final Thoughts

• Integration of PE into the fabric of the
  organization.
• Inclusive style of participation on design and
  implementation teams. People need to feel
  involved -- not an audit function, but part of a
  continuous improvement strategy for the agency.
• Clear communication of findings and implications
  for all levels of the organization.
• Go beyond the analysis of passive data to design
  and carry out experimental and quasi-experimental
  research on specific interventions --
  evidence-based practices.

19
Final Thoughts

• Seek out other program Evaluation Specialists in
  State Agencies and other human service
  institutions to share ideas, problems and
  challenges.
• The PE Summit here in Salt Lake City is a
  wonderful start to develop a learning community
  among professionals with similar roles and
  responsibilities throughout the nation.
• It is my hope that this group will continue
  meeting and communicating in the future.
• Best of luck to all of you!!