Performance Indicators:

1
Performance Indicators

• Development of a Methodology for Reporting
  Student Outcomes in a Provincial System
• Presentation by Thelma G. Lussier
• CIRPA Conference Montreal October 2004

2
Outline

• The requirement for accountability and
  performance measures in post secondary
  institutions in Manitoba
• The reasons for it
• How the Ministry and the post secondary
  institutions were able to engage in a productive
  process
• Presentation of U of M results

3
Timeline

• In June 1999, Council on Post-Secondary Education
  (COPSE) invited institutions to a presentation on
  PIs and Accountability Measures in other
  jurisdictions.
• Asked institutions to present what PIs and
  accountability measures they use
• Asked them to come prepared to discuss next steps

4
At That Meeting

• Described reasons for PIs as
• A general desire on the part of the public,
  government, parents, and industry to have more
  information about what institutions do and why
  whether they spend their money appropriately,
  what is the value of the education offered the
  extent to which programs respond to labour market
  needs, and student needs whether research
  outcomes make a contribution to national RD
  efforts, and whether institutions have a climate
  of responsiveness.

5
Results

• Institutions agreed to appoint a representative
  to a working group at U of M this was our
  Vice-Provost (Programs) and he asked me to also
  attend meetings
• Met again in the fall reiteration of previous
  material

6
The Next Two Years

• Drafts exchanged back and forth
• On one hand COPSE saying they wanted PIs, for
  example Measures for teaching
• In response, institutions asking what is your
  objective and what will you do with the
  information once you have it?

7
Election - Renewed Mandate

• With that new mandate, Department of Advanced
  Education published a paper on priorities with a
  number of goals
• Efforts were now clearly focused
• The one relevant to this discussion was improve
  the success rate in all programs and
  institutions.

8
Priorities related to this goal

• Gather better data on student retention
• Gather data on a range of outcomes measures such
  as graduation, employment, etc.
• Renew and strengthen Access programs
• Others related to student finance

9
Clear expectations set

• COPSE decided to narrow focus to student outcomes
  in order to achieve the required results in a
  timely fashion
• Institutions asked to review the framework and
  nominate a representative on the working group
  for this express purpose
• U of M appointed me

10
Results of the 1st Meeting

• Assurance was given that the indicators would not
  be linked to funding
• In order to have data in a timely fashion, the
  first indicator would be data on student
  graduation, attrition, and continuation

11
Results of 1st Meeting Cont

• Colleges and Universities would need different
  methodologies
• Reporting would be by program, because of the
  variation in scope of programs
• Need multi year data single year anomalies
  especially small programs

12
Results of 1st Meeting Cont

• Institutions would be involved in the
  interpretation and contextualization of results
• COPSE cautioned on public reaction to actual drop
  out rates
• COPSE to get benchmarking data for context

13
Definitions

• Only undergraduates, only full-time (60)
• Outcomes for each cohort will be reported on at
  the end of 1, 5, and 7 years
• Co-op programs at end of 1, 6, and 8 years
• Only students registered on the census reporting
  date of November 1

14
Definition of Categories of Programs

• Direct Entry also First Entry admitting
  directly from high school or equivalent
  University 1, Music, Engineering
• Indirect Entry also Second Entry requires
  previous university study for admission Law,
  Medicine, Mgmt.
• Co-op - subset of indirect program must have
  paid work component

15
Cohort is an issue for U of M

• In most institutions, entering the University and
  entering a program are the same thing
• But at the U of M, we have a common first year
  called University 1
• You cant graduate from University 1
• However, COPSE wanted institutions to report
  outcomes with faculty breakdowns within each group

16
Determining first and second entry

• So we have to track University 1 students and
  allocate them to whatever faculty and category of
  program they enter.

17
Other definition issues

• Students transferring to other institutions not
  really attrition. Convinced COPSE to use results
  of surveys to use an estimate of transfers
• Students who dont perform well this is not
  necessarily the institutions fault provides
  additional context

18
Inter Institutional Issues

• Other institutions do not have a University 1
  program drop out rates for certain programs may
  be higher
• Balanced by the fact that reporting is by
  program, not at the aggregate institutional
  level. The high graduation rates of Law and
  Medicine boosts the U of Ms overall graduation
  rate relative to others

19
Overview of the Process

• Took about two years of work and countless
  meetings
• Complicated by the fact that U of M could only
  start at 1998 with the introduction of University
  1.
• We collaboratively developed a methodology
  document (draft attached in handout)

20
Final Agreed List of Outcomes for Cohorts

• Graduated
• Continuing
• Continuing/Graduated at another Post Secondary
  Institution (estimated based on surveys)
• Attrition
• Ineligible to proceed

21
Limitations of data

• Because of the 5 and 7 year time lapse, still
  limited data
• Trying to make the University 1 match the more
  typical Arts/Science approach confuses even us
• Complicated to represent in a meaningful way

22
University of Manitoba data

• The next slides will show the data for the U of M
  for direct, non direct and co-op programs
• It is for students who enrolled between Sept.
  1998 and April 1999 and shows their status at the
  beginning of the next year

23
University of Manitoba Student OutcomesStudents
Starting Between Sept. 1998 - April 1999
24
University of Manitoba Student OutcomesStudents
Starting Between Sept. 1998 - April 1999
25
University of Manitoba Student OutcomesStudents
Starting Between Sept. 1998 - April 1999
26
What did we learn from this process?

• Develop your expertise anticipate requests
• Develop your relationships with your Ministry
  staff like you, they are just trying to do
  their jobs
• Share your expertise it makes for a win-win
  situation. Otherwise, a methodology may be
  imposed

27
Questions/Discussion

• The intention is that the collaboration among the
  institutions and COPSE will continue for other
  indicators