Program Reviews: An Essential Part of the Divisional Assessment Plan

1
Program Reviews An Essential Part of the
Divisional Assessment Plan

• Assessment Workshop
• Fall 2007
• Presented by Susan Martin, Ed.D.

2
Program Description and Intended outcomes

• ABCs of Program Review and Using the CAS
  Standards Program reviews are a fundamental
  component of our divisional assessment plan.
  After this session you should be able to
• Provide colleagues with a simple definition of
  what a program review is
• Name and describe the steps of the program review
  process
• Explain what the CAS Standards are and how they
  are used in the program review process.

3
Interview your partner and introduce her/his
responses to the following

• Have you ever conducted a program review or
  program evaluation? If so describe. If not
  discuss what you know about this process.
• What professional or service standards exist for
  your department?
• What do you already know about CAS?
• What do you want to learn today? What burning
  question do you want answered?

4
Assessment is the systematic collection, review,
and use of information (evidence and data) about
educational programs and services undertaken for
the purpose of improving or increasing student
learning or the quality of service delivered.
(Adapted from Marchese, 1998)
Types of Assessment Activities
Created by Gavin Henning, Dartmouth College,
2005. Used with Permission.
5
What is a program review?

• A systematic process of collecting and analyzing
  data/evidence about a program and comparing these
  to some standard for the purposes of evaluating
  effectiveness, impact, and/or efficiency
• EffectivenessLooks at program objectives and
  asks whether they have been achieved (or not)
• Impactthe degree to which the program has
  resulted in changes
• Efficiencyresult to which the program has been
  productive with regard to resources
• Source Boulmetis, J. Dutwin, P. (2005). The
  ABCs of Evaluation, 2nd ed. San Francisco
  Jossey Bass.

6
Why must we do program reviews?

• Internal Reasons to Division of Student Affairs
• To be the best we can be Quality assurance and
  improvement
• Better resource allocation To inform planning
  and budget processes
• Essential part of holistic approach to
  assessment An assessment essential that is part
  of our Divisional Assessment Plan
• External Reasons
• Program Reviews will be part of UMBCs assessment
  approach
• Middle States expects/recognizes CAS
• Enhance credibility, value and worth to outside
  constituencies
• Funding source requirements

7
On what cycle are we doing program reviews?
8
Primary Questions(insert department)

• What should ________do?
• Does_________ do what it is supposed to do?
• How have students using ______ change?
• What is working well in ______?
• What needs to be improved in _____?
• What actions should be taken (by whom and by
  when) to improve ___________?

9
Program Review Frameworks

• Accreditation processes
• Health Services Accreditation Association for
  Ambulatory Health Care The Joint Commission
• Counseling Services International Association of
  Counseling Services http//iacsinc.org
• Professional Association Statements
• ACUHO-I Standards and Ethical Principles for
  College and University Housing Professionals
• CAS Professional Standards for Higher Education
  http//www.cas.edu/
• Campus Activities Campus Information Services
• Student Leadership Programs
• Multicultural Student Programs
• Student Conduct Programs

10
Basic Steps of A Program Review

• AHEAD OF TIME Decide on Purpose What is the
  focus of the evaluation? What question(s) will be
  answered and why? (Director and program review
  team chair)
• Pick and train team Who will you involve? Lay
  groundwork for team interaction (Director and
  chair)
• Choose and review specific standards to be
  examined CAS, IACS, ACHUHO? (Director, chair,
  and discuss with team)
• Determine data sources and assemble these
• ASSESS PROGRAM
• Assign ratings to determine strengths, weaknesses
• Summarize findings
• Identify action plan
• __________________________________________
• Afterwards Unit implements action plan
• Cycle repeated in 5 years

11
CAS
Council for the Advancement of Standards in
Higher Education

• Founded in 1979
• A consortium of 36 professional organizations
  comprising a constituency of over 100,000
• Member associations send representatives to the
  CAS Board of Directors
• Consensus-oriented, collaborative
• Focused on quality programs and services for
  students
• Guide practice by student affairs, student
  development, and student support service providers

12
Essential Elements in the CAS Approach

• An institutional culture that values
  involvement of all its members in
  decision-making
• Quality indicators that are determined by
  the institution
• Use of standards guidelines in quality
  assurance initiatives

13
CAS Resources

• 35 functional area standards (pub. in 6th
  edition, 2006)
• Reviewed and revised regularly
• General standards contained within every other
  set of standards
• Self-Assessment Guides, with instructions
  training
• Frameworks for Assessing Learning Development
  Outcomes (FALDOs)
• Masters level student affairs administration
  preparation program standards
• Contextual statements for each area
• CAS Statement of Shared Ethical Principles
• CAS Characteristics of Individual Excellence

14
Getting the Self-Assessment Process Started
Appoint and Train the Self-Assessment Team

• Single functional area calls for 3-5
• Include members from the outside (students,
  faculty, other constituents with stake)
• Establish team ground rules
• Establish teams inter-rater reliability
• Encourage team discussion and expect team members
  to disagree

15
Team Actions to Conduct the Assessment Process

• Finalize Standards to be examined Be sure all
  team members understand how these will be rated
• Gather and analyze relevant quantitative and
  qualitative data
• Individuals rate each and every criterion measure
• Obtain additional documentary evidence if
  required to make an informed team decision
• Complete the assessment, ratings, action plan
  worksheets

16
Compile Review Evaluative Evidence
K E Y T O A S S E S S M E N T The
self-assessment is not complete until relevant
data and related documentation are in place to
support the raters
judgments.
17
Examples of Evaluative Evidence

• STUDENT RECRUITMENT MATERIALS
• Brochures other program information
• Participation policies procedures
• PROGRAM DOCUMENTS
• Mission statements program purpose philosophy
  statements
• Catalogs related materials
• Staff student manuals policies procedures
  statements
• ADMINISTRATIVE DOCUMENTS
• Organization charts student staff profiles
• Financial resource statements budgets
• Annual reports
• STAFF ACTIVITY REPORTS
• Curriculum vitae resumes professional activity
• Service to other programs, departments, or
  community
• STUDENT ACTIVITY REPORTS
• Portfolios, developmental transcripts, resumes
• Reports of student service
• RESEARCH EVALUATION DATA
• Needs assessments self-studies

18
Quantitative Data

• Summaries of objective responses on a
  questionnaire or program evaluation
• Statistics about use by students and various
  population subgroups
• Needs assessments, follow-up studies, and
  self-study reports
• Institutional research reports and fact books

19
Qualitative Data

• Focus group information
• Written summaries of responses to open-ended
  questions in interviews and on evaluations
• Client satisfaction surveys, self-reports, and
  written comments, both solicited and unsolicited

20
Self Assessment Guides (SAGS) offer a ready
format for evaluation

• In most instances, there are multiple criterion
  statements for each standard
• Each criterion measure focuses on a particular
  aspect of the standard, allowing raters to
  express more detailed and specific judgments

21
Assessment Criteria Example
Part 6 FINANCIAL RESOURCES Counseling Services
(CS) must have adequate funding to accomplish its
mission and goals. Funding priorities must be
determined within the context of the stated
mission, goals, objectives, and comprehensive
analysis of the needs and capabilities of
students and the availability of internal or
external resources. CS must demonstrate fiscal
responsibility and cost effectiveness consistent
with institutional protocols. ND 1
2 3
4 NR Not Done Not Met Minimally Met
Well Met Fully Met Not Rated PART 6.
FINANCIAL RESOURCES (Criterion Measures)
Rating Scale NOTES 6.1 The program has adequate
funding to accomplish its mission and goals. ND
1 2 3 4 NR 6.2 Funding priorities are
determined within the context of program
mission, student needs, and available
fiscal resources. ND 1 2 3 4
NR 6.3 The program demonstrates fiscal
responsibility and cost effectiveness
consistent with institutional protocols. ND
1 2 3 4 NR Part 6 Financial Resources
Overview Questions A. What is the funding
strategy for the program? B. What evidence
exists to confirm fiscal responsibility and
cost-effectiveness?
22
Team Actions to Conduct the Assessment Process

• For each of the 13 Parts, identify the criterion
  measure item number(s) for which there is
  substantial rating discrepancy.
• Items not circled should reflect consensus among
  raters that practice in that area is
  satisfactory.
• Items where judgment variance occurs need to be
  discussed thoroughly by team members.
• Follow this action by determining which practices
  can be designated as excellent or
  unsatisfactory.
• List the items requiring follow-up action,
  including any criterion measure rated as
  unsatisfactory by any reviewer.
• Complete the Overview Questions at the end of
  each Part.

23
Action Plan Components

• Program Action Plans
• Identify responsible parties to complete the
  action steps timeline
• Summary action plan creates an overview of the
  work to be done

PLAN For the Future
24
Questions about program reviews?