Using CAS Standards and Review Process for Improving Student Learning and Program Outcomes

1
Using CAS Standards and Review Process for
Improving Student Learning and Program Outcomes

• Patricia Carretta
• George Mason University
• CAS Officer-at-Large
• NACE Representative to CAS
• NASPA 2008 International Assessment Retention
  Conference

2
Purpose

• CAS Standards and Review Process
• Integrating CAS into assessment of programs and
  services and student learning outcomes

3
Role and Use of CAS Standards

• Student Affairs / Student Services at the
  division level?
• Student Affairs / Student Services at the
  departmental or program level?
• Other?
• Have looked at the CAS Standards for at least one
  functional area?
• Have participated in a department/program review
  process using CAS standards?
• Have used CAS standards for other purposes?

4
Participant Outcomes

• Define the role of CAS self-assessment / review
  process within a comprehensive assessment plan
• List the seven step process of conducting a CAS
  review
• Appreciate the value of engaging a variety of
  institutional stakeholders in the review process
• Strategize how to conduct a comprehensive
  assessment of programs, services and student
  learning using CAS Standards
• Formulate an assessment plan
• Understand the importance of assessing and
  reporting on input data as well as student
  learning and development outcomes as a strategies
  for improving learning and program quality

5
About CAS
The mission of the Council for the Advancement
of Standards in Higher Education (CAS) is to
promote the improvement of programs and services
to enhance the quality of student learning and
development. CAS is a consortium of
professional associations who work
collaboratively to develop and promulgate
standards and guidelines and to encourage
self-assessment.
6
CAS Resources

• CAS Professional Standards, 2006 edition
• Self-Assessment Guides
• CAS Statement of Shared Ethical Principles
• CAS Characteristics of Individual Excellence
• Frameworks for Assessing Learning and Development
  Outcomes

7
CAS Guiding Principles

• Students and Their Institutions
• Diversity and Multiculturalism
• Organization, Leadership, and Human Resources
• Health Engendering Environments
• Ethical Considerations

8
Organization of CAS Standards

• Mission
• Program
• Leadership
• Human Resources
• Ethics
• Legal Responsibilities
• Equity and Access
• Diversity

• Organization and Management
• Campus and External Relations
• Financial Resources
• Technology
• Facilities and Equipment
• Assessment and Evaluation

9
Student Learning Development

• Must promote student learning and development
  outcomes
• Must identify relevant and desirable student
  learning development outcomes
• Must assess outcomes
• Must provide evidence of their impact on student
  learning and development
• Must articulate how they contribute to or support
  student learning and development

10
CAS Student Learning Development Outcomes

• Knowledge acquisition, construction, integration
  and application
• Cognitive complexity
• Intrapersonal development
• Interpersonal competence
• Humanitarianism civic engagement
• Practical competence

11
Student Learning Development Outcomes

• Consistent with the institutional mission,
  programs and services must identify relevant and
  desirable student learning and development
  outcomes from among the six domains. When
  creating opportunities for student learning and
  development, programs and services must explore
  possibilities for collaboration with faculty
  members and other colleagues.
• Programs and services must assess relevant and
  desirable student learning and development
  outcomes and provide evidence of their impact on
  student learning and development. Programs and
  services must articulate how they contribute to
  or support student learning and development in
  the domains not specifically assessed.

12
Essential Elementsin the CAS Approach

• Culture that values involvement of all its
  members
• Quality indicators
• Use of standards and guidelines
• Willingness/capacity
• To examine itself and assemble results
• To report and use the results

13
Uses of CAS Standards

• Credibility, accountability, improvement
• Program service improvement measures of
  quality and effectiveness measures of impact on
  learning
• Design of new programs services
• Restructuring / reorganization
• Change in leadership
• Institutional self-studies
• Preparation for accreditation
• Staff development
• Academic preparation

14
CAS Standards and Guidelines

• STANDARD (BOLD TYPE AUXILARY VERB MUST)
• To support the institutions mission and goals,
  Career services must provide faculty and staff
  and administrative units with information,
  guidance, and support on career development and
  employment issues and linkages with the broader
  community.
• GUIDELINE (LIGHT-FACED TYPE VERB SHOULD)
• Career Services should support faculty and staff
  and administrative units by
• Identifying and disseminating information on
  employment trends, top employing organizations,
  and co-op and internship sponsors.

15
Comprehensive Assessment Plan

• Includes
• Assessment of programs and services
• Assessment of student outcomes, including student
  learning and development

16
Program Evaluation

• Is the program or service functioning effectively
  to achieve its mission?
• What evidence is available to support the
  determination?
• How is evidence used to make program decisions?

17
Student Outcomes Assessment

• What is the effect of our work on students?
• How are they different as a result of interacting
  with our programs and services?
• How do we demonstrate their learning?
• How do we know? Measure?

18
The CAS SAG A Self-Assessment Guide

• Translates CAS standards into an effective
  workbook / evaluative format
• Promotes program self-assessment and development
• Informs on program strengths and weaknesses
• Supports professional staff development
• Leads to enhanced programs and services that
  benefit student learning and development

19
CAS Self-Assessment Process

• Establish and prepare the Self-Assessment Team
• Initiate the Self Study
• Compile and review Documentary and Evaluative
  Evidence
• Identify Discrepancies
• Determine appropriate Corrective Action
• Recommend Steps for Program Enhancement
• Prepare an Action Plan

20
The Self-Assessment Team

• Appoint and Train the Team
• Size of team
• Composition internal, external
• Coordinator/Leader
• Establish team ground rules
• Discuss meaning of each standard
• Establish teams inter-rater reliability
• Encourage team discussion expect disagreements
  commit to consensual resolution

21
Team Actions

• Decide whether to include guidelines or other
  measures that go beyond the standards
• Gather and analyze quantitative and qualitative
  data
• Individuals rate each and every criterion measure
• Obtain additional documentary evidence if
  required to make an informed team decision
• Complete the assessment, ratings action plan
  worksheets

22
Evaluative Evidence

• STUDENT RECRUITMENT MATERIALS
• Brochures other program information
• Participation policies procedures
• PROGRAM DOCUMENTS
• Mission statements program purpose philosophy
  statements
• Catalogs related materials
• Staff student manuals policies procedures
  statements
• ADMINISTRATIVE DOCUMENTS
• Organization charts student staff profiles
• Financial resource statements budgets
• Annual reports
• STAFF ACTIVITY REPORTS
• Curriculum vitae resumes professional activity
• Service to other programs, departments, or
  community
• STUDENT ACTIVITY REPORTS
• Portfolios, developmental transcripts, resumes
• Reports of student service
• RESEARCH EVALUATION DATA
• Needs assessments self-studies

23
Quantitative Data

• Summary of objective responses on a questionnaire
  or program evaluation
• Statistics about use by students and various
  population subgroups
• Needs assessments, follow-up studies, and
  self-study reports
• Institutional research reports and fact books
• Comparative or benchmark data at local, regional
  or national level

24
Qualitative Data

• Focus group information
• Written summaries of responses to open-ended
  questions in interviews and on evaluations
• Client satisfaction surveys, self-reports, and
  written comments, both solicited and unsolicited

25
Using the SAGs

• SAGs offer a ready format for evaluation
• In most instances, there are multiple criterion
  statements for each standard
• Each criterion measure focuses on a particular
  aspect of the standard, allowing raters to
  express more detailed and specific judgments

26
Criterion Measure Rating Scale

• ND 1 2 3 4 NR
• Not Done Not met Minimally Well
  Fully Not Rated
• Met Met Met

ND 1 2 3 4 NR Not Done Not met
Minimally Well Fully Not
Rated Met Met Met

• Using this scale, consider each criterion
  statement and decide
• The extent to which each criterion measure has
  been met by the program or service

27
Assessment Criteria Example

• Part 6 FINANCIAL RESOURCES
• Counseling Services (CS) must have adequate
  funding to accomplish its mission and goals.
  Funding priorities must be determined within the
  context of the stated mission, goals, objectives,
  and comprehensive analysis of the needs and
  capabilities of students and the availability of
  internal or external resources.
• CS must demonstrate fiscal responsibility and
  cost effectiveness consistent with institutional
  protocols.
• ND 1 2 3
  4 NR
• Not Done Not Met Minimally Met Well
  Met Fully Met Not Rated
• PART 6. FINANCIAL RESOURCES (Criterion
  Measures) Rating Scale NOTES
• 6.1 The program has adequate funding to
  accomplish its mission and goals. ND 1 2 3
  4 NR
• 6.2 Funding priorities are determined within
  the context of program mission,
• student needs, and available fiscal
  resources. ND 1 2 3 4 NR
• 6.3 The program demonstrates fiscal
  responsibility and cost effectiveness
• consistent with institutional protocols. ND
  1 2 3 4 NR
• Part 6 Financial Resources Overview Questions

28
CAS Work Form AAssessment, Ratings, and
Significant Items
CAS Work Form A Assessment, Ratings, and
Significant Items INSTRUCTIONS This work form
should be completed following individual ratings
of the participants. For each of the 13 Parts,
identify (circle) the criterion measure item
number(s) in the column labeled for which there
is a substantial rating discrepancy (two or more
ratings apart). Items not circled should reflect
consensus among judges that practice in that area
is satisfactory. Items where judgment variance
occurs need to be discussed thoroughly by team
members. Follow this action by determining which
practices (criterion measures) can be designated
as excellent or unsatisfactory and record
them in the Step One column. In Step Two, list
the items requiring follow-up action including
any criterion measure rated as being
unsatisfactory by any reviewer.
29
CAS Work Form A continued
30
CAS Work Form B Follow Up Actions

• INSTRUCTIONS
• The purpose of this work form is to begin the
  planning for action to be taken on practices
  judged to merit follow-up (See Step 1, Work Form
  A). In Step Three, transfer short descriptions of
  the practices requiring follow-up and detail
  these items using the table format provided.
• Step Three Describe the current practice that
  requires change and actions to initiate the
  change

31
CAS Work Form CSummary Action Plan

• Step Four
• This form concludes the self-assessment process
  and calls for action to be taken as a consequence
  of study results. Write a brief action plan
  statement in the spaces below for each Part in
  which action is required.

32
Closing the loop

• Purpose scope of study is defined
• Self-study team is trained
• Self-study is conducted
• Evaluative evidence is collected
• Discrepancies are identified
• Appropriate actions are determined
• Special actions for program enhancement are
  recommended
• Action plan is developed and communicated

33
Student Outcomes Assessment

• What is the effect of our work on students?
• How are they different as a result of interacting
  with our programs and services?
• What have students learned?
• How have they developed?

34
Why Focus on Student Learning?

• Accountability
• Alignment with institutions mission
• Consistent with language of the academy
• Quality improvement
• More effective learning
• Better defined and more effective planning and
  strategy development

35
Defining Learning Outcomes

• Outcomes usually identify growth in some
  dimension of knowing, being or doing.
• Susan Komives Sarah Schoper, Developing
  Learning Outcomes,
• Learning Reconsidered 2 (2006)
• Knowledge (cognitive)
• Attitude (affective)
• Skill (psychomotor)

36
CAS Student Learning Development Outcomes

• Knowledge acquisition, construction, integration
  and application
• Cognitive complexity
• Intrapersonal development
• Interpersonal competence
• Humanitarianism civic engagement
• Practical competence

37
Plan for Assessing Student Learning Outcomes

• Review/revise mission
• Identify major services, programs, activities
• Specify desired student outcomes
• Identify information needed to show student
  learning
• Identify existing and new data needed
• Develop process for measuring outcomes
• Identify ways to use findings to change/improve

38
Writing Learning Outcomes

• Identify words/phrases that describe what an
  ideal program participant or college graduate
  would be able to
• Know Be Do as a result of engaging with
  your program, activity or service.
• What would he/she learn from the experience?

39
Student Outcomes Matrix
40
Writing Learning Outcomes

• Describe the learner characteristics of
  users/participants of programs and services. What
  knowledge, skills, experiences do they start
  with?
• Determine acceptable levels of performance
• Consider how you would provide evidence of
  achievement

41
Ways to measure learning outcomes

• Surveys, questionnaires, self-report (indirect)
• Observations
• Interviews
• Focus groups
• Reflective journals
• Performance evaluations
• Pre-test/post-test
• Portfolios
• Capstone projects
• Review of project/performance by peers and
  internal/external reviewers

42
Tips, suggestions

• Identify all your program outcomes
• Rank outcomes based on importance
• Activity outcome
• Program outcome
• Institutional or division outcome
• Identify where learning opportunities are
  occurring (mapping)
• Design and implement assessment cycle one
  outcome at a time

43
Remember

• Start small
• Start doable
• Start narrow
• Start targeted
• Start!
• One program, one workshop, one desired outcome,
  one measured tool

44
Additional CAS Resources

• www.cas.edu
• Standards book
• Prologue Context with How to information
• FALDOs book and CD formats as package
• SAGs CD or individually