The CAS Basics A Training in the Assessment Process

1
The CAS BasicsA Training in the Assessment
Process

• For Messiah Collegeby
• Christina Anastasi
• 5/22/07

2
Introductions

• Name
• Program Representing
• Personal Learning Objectives

3
What is Assessment?

• Your Ideas?
• Systematic collection and interpretation of data
• Locally designed and executed

4
Upcraft SchuhAssessment in Student Affairs
(1996)

• Assessment Any effort, analyze, and interpret
  evidence which describes institutional,
  departmental, divisional, or agency effectiveness
• Evaluation Any effort to use assessment
  evidence to improve institutional, departmental,
  divisional, or agency effectiveness

5
Why Do Assessments?

• Your Ideas?
• Preparation for Accreditation
• Licensure Certification
• Program Review
• Measures of program service effectiveness
• Institutional self-studies
• Design of new programs services
• Staff development
• Academic preparation
• Credibility and accountability

6
Why Do Assessments?

• To improve education and student learning
• To examine congruence with institution's stated
  purpose
• Performance funding
• To address concerns of effectiveness (access,
  cost, and decline in student learning)
• Ethical practice
• Curiosity

7
Advantages of Self-Assessment

• Internally Driven
• Supports Staff Development
• You are in the Best Position to Evaluate Your
  Programs and Services
• Recognition and Rewards are Made Locally
• Develops a Shared Vision Among Various
  Constituents

8
History of CAS

• Founded in 1979 in response to efforts to improve
  program standards for universities offering a
  masters degree in student affairs administration
• Council for the Advancement of Standards for
  Student Services / Development Programs
• Initially a Study Group of ACPA NASPA
• First standards published in 1986 American
  College Testing (ACT) Grant
• 1992 Council for the Advancement of Standards
  in Higher Education

9
CAS MissionLet us raise a standard to which the
wise and honest can repair.George Washington,
1787

• Establish, adopt, and disseminate unified and
  timely professional standards to guide student
  learning and development programs and services
• Promote assessment and improvement of higher
  education programs and services through
  self-study, evaluation, and the use of CAS
  Standards
• Promote encourage a focus on quality assurance
• Promote inter-association efforts to address
  these issues

10
CAS Principles- Students and Their Institutions-

• Student is considered as a unique, whole person
• Institutional environments shape learning
• Responsibility for learning rests with the
  student
• Institutions provide opportunities for learning
• Institutions reflect society and its diversity

11
CAS Principles- Diversity and Multiculturalism -

• Institutions embrace diversity and eliminate
  barriers that impede student learning
• Justice and respect for differences
  bondindividuals to community

12
CAS Principles- Organization, Leadership, and
Human Resources -

• Leadership is essential for institutionalsuccess
• Institutional success is related to clarity
  ofmission
• Qualifications of staff members is tieddirectly
  to quality of educational programsand services
• Leaders possess sound educationalpreparation and
  experience

13
CAS Principles- Health Engendering Environments
-

• Educational programs and services prosper in
  benevolent environments that provide students
  with appropriate levels of challenge and support.

14
CAS Principles- Ethical Considerations -

• Educational service providers provideimpeccable
  ethical behavior in theirprofessional and
  personal lives.

15
CAS Purpose- Philosophy Beliefs -

• Excellence in Higher Education
• Collaboration Between Teacher Learner
• Ethics in Educational Practice
• Student Development As a Major Goal of Higher
  Education
• Student Responsibility for Learning

16
Who is CAS?

• A consortium of 36 professional organizations
  comprising a constituency of over 100,000
• Joins Academic Affairs, Student Affairs, and
  Service Units into One Council
• 2004 was its 25th Anniversary
• Member associations send representatives to the
  CAS Board of Directors
• Consensus-oriented, collaborative

17
Who is CAS?- Member Associations -

• Associations from both U.S. Canada
• See List from www.cas.edu

18
Who is CAS?- Executive Committee -

• Executive Director Phyllis Mable Council for
  the Advancement of Standards in Higher Education
• President Jan ArminioAssociate Professor and
  Chair, Department of Counseling and College
  Student Personnel Shippensburg University
• Secretary Douglas Lange Association of
  Fraternity Advisors
• Treasurer Carmen Guevara Neuberger Public
  Director
• Officer-at-Large Laura DeanVice President
  Dean of Student DevelopmentPeace College
• Officer-at-Large Paula SwinfordDirector, Health
  Promotion and Prevention ServicesUniversity of
  Southern California
• Officer-at-Large Susan KomivesAssociate
  Professor, Counseling Personnel Services
  University of Maryland - College Park

19
Who is CAS?- Board of Directors -

• 2 Representatives from Each Member Association
  (Director / Alternate Director)
• 1 Representative Attends Meetings
• Directory
• Meetings

20
CAS Resources

• 35 functional area standards (pub. in 6th
  edition, 2006)
• Reviewed and revised regularly
• General standards contained within every other
  set of standards
• Masters level student affairs administration
  preparation program standards
• Contextual statements for each area
• Self-Assessment Guides (with instructions
  training)
• Frameworks for Assessing Learning Development
  Outcomes (FALDOs)
• CAS Statement of Shared Ethical Principles
• CAS Characteristics of Individual Excellence

21
CAS Standards

• Academic AdvisingAdmission ProgramsAlcohol,
  Tobacco, and Other Drug ProgramCampus
  ActivitiesCampus Information and Visitor
  ServicesCampus Religious and/or Spiritual
  ProgramsCareer ServicesClinical Health
  ServicesCollege Honor SocietiesCollege
  UnionCommuter and Off-Campus Living
  ProgramsConference and Event ProgramsCounseling
  ServicesDisability Support ServicesDistance
  Education ProgramsEducation Abroad Programs and
  ServicesFinancial Aid ProgramsFraternity and
  Sorority AdvisingHealth Promotion
  ServicesHousing and Residential Life
  ProgramsInternational Student Programs and
  ServicesInternship ProgramsLearning Assistance
  ProgramsLesbian, Gay, Bisexual, and Transgender
  Programs and ServicesMasters-Level Graduate
  Program for Student Affairs ProfessionalsMulticul
  tural Student Programs and ServicesOrientation
  ProgramsOutcome Assessment and Program
  EvaluationRecreational Sports ProgramsRegistrar
  Programs and ServicesService-Learning
  ProgramsStudent Conduct ProgramsStudent
  Leadership ProgramsTRIO and Other Educational
  Opportunity ProgramsWomen Student Programs and
  Services

22
CAS Standards Format 13 Component Parts

• 1. Mission
• 2. Program
• 3. Leadership
• 4. Organization management
• 5. Human resources
• 6. Financial resources
• 7. Facilities, technology, equipment
• 8. Legal responsibilities
• 9. Equity and access
• 10. Campus and community relations
• 11. Diversity
• 12. Ethics
• 13. Assessment evaluation

23
Program Learning and Development Outcome Domains

• Intellectual growth
• Effective communication
• Enhanced self-esteem
• Realistic self-appraisal
• Clarified values
• Career choices
• Leadership development
• Healthy behavior
• Meaningful interpersonal relationships

• Independence
• Collaboration
• Social responsibility
• Satisfying and productive lifestyles
• Appreciating diversity
• Spiritual awareness
• Personal and educational goals

24
Understanding CAS Standards

• CAS standards . . .
• Represent indispensable requirements of
  practice
• Are achievable by any and all programs of
  quality
• Appear in bold print
• Use auxiliary verbs must shall

25
Understanding CAS Guidelines

• CAS guidelines . . .
• Clarify amplify standards
• Guide enhanced practice beyond essential
  functions
• Appear in light-faced type
• Use verbs should may

26
Standard Guideline Example
STANDARD BOLD TYPE AUXILIARY VERBS MUST
SHALL Counseling services must be a) intentional,
b) coherent, c) based on theories and knowledge
of counseling, learning, and human development,
d) reflective of developmental and demographic
profiles of the student population, and e)
responsive to the needs of individuals within a
higher education setting. GUIDELINE LIGHT-
FACED TYPE AUXILIARY VERBS SHOULD
MAY Counseling services should provide
consultation, supervision, and in-service
professional development for faculty members,
administrators, staff and student staff members,
and paraprofessionals.




27
The CAS SAG A Self-Assessment Guide

• Translates CAS standards into an effective
  workbook format
• Promotes program self-assessment and development
• Informs on program strengths and weaknesses
• Supports professional staff development
• Leads to enhanced student learning and
  development

28
Frameworks for Assessing Learning and Development
Outcomes

• FALDOs companion to standards book
• Provide Clarity on Learning Domains
• Guide Assessment Activities with Questions
  Instruments
• Chapters for each learning outcome domain with a
  theoretical description of the domain (e. g.,
  leadership development, social responsibility,
  career choices), assessment examples, list of
  possible instruments, and additional resources
• Published as Book CD set

29
CAS Approach to Self-Regulation and
Self-Assessment

• Fundamental Elements
• 1. Quality best determined by institution's
  people2. Goals based on shared vision3.
  Nationally agreed-upon quality indicators4.
  Reliance on honesty with meticulous evaluation5.
  Assembling of results into a plan for improvement

30
Ground Rules for Quality Self-Assessment

• A commitment by the management team to improve
  the program or service
• Mutual trust among the members (confidentiality)
• Decisions would always be made that are in the
  best interest of the college or university
• All team members have something of value to say
  and can contribute to the decisions of the group
• Programs or services should be recognized and
  rewarded if they are exceptional in surpassing
  the standards and guidelines

31
Comprehensive Assessment Model- Upcraft Schuh
(1996) -

• Use of Services, Programs, Facilities
• Student Needs
• Student Satisfaction
• Campus Environments
• Student Cultures
• Program Service Outcomes
• Benchmarking Comparing Performance Across
  Organizations
• Measuring Effectiveness Against Professional
  Standards

32
Good Assessments

• Provide useful information
• Provide reasonably accurate information
• Protect privacy of students
• Are systematic and regular
• Are effective in terms of time, cost, and other
  resources

33
CAS Self-Assessment Process

• 1) Establish and Prepare Self-assessment Team
• 2) Understand the CAS Standards Guidelines
• 3) Compile and Review the Evidence
• - Gathering Evidence (Quantitative
  Qualitative)
• - Conduct Rating
• - Gather Consensus
• - Identify and Summarize Evaluative Evidence
• 4) Judge Performance
• - Identify Strengths Discrepancies
• 5) Write the Action Plan
• - Determine Appropriate Corrective Action
• - Recommend Steps for Program Enhancement
• - Prepare an Action Plan

34
1 - Establish Prepare Self-Assessment Team

• Division-wide requires 8-10 members
• Single functional area calls for 3-5
• Include members from the outside
• Establish team ground rules
• Establish teams inter-rater reliability
• Encourage team discussion and expect team members
  to disagree

35
1 - Establish Prepare Self-Assessment
Team(Team Actions to Conduct the Assessment
Process)

• Decide whether to include guidelines or other
  measures that go beyond the standards
• Gather and analyze relevant quantitative and
  qualitative data
• Individuals rate each and every criterion measure
• Obtain additional documentary evidence if
  required to make an informed team decision
• Complete the assessment, ratings, action plan
  worksheets

36
1 - Establish Prepare Self-Assessment
Team(Using Guidelines as Standards)

• Write criterion statements and add these to the
  teams scoring scheme.
• Exclude criterion measures deemed not
  applicable to the program only on the very rare
  occasion when an extenuating circumstance exists
  (e.g., an institution that is not allowed by
  state law to follow affirmative action
  procedures.)

37
3 Compile Review the Evidence
K E Y T O A S S E S S M E N T The
self-assessment is not complete until relevant
data and related documentation are in place to
support the raters
judgments.
38
Examples of Evaluative Evidence

• STUDENT RECRUITMENT MATERIALS
• Brochures other program information
• Participation policies procedures
• PROGRAM DOCUMENTS
• Mission statements program purpose philosophy
  statements
• Catalogs related materials
• Staff student manuals policies procedures
  statements
• ADMINISTRATIVE DOCUMENTS
• Organization charts student staff profiles
• Financial resource statements budgets
• Annual reports
• STAFF ACTIVITY REPORTS
• Curriculum vitae resumes professional activity
• Service to other programs, departments, or
  community
• STUDENT ACTIVITY REPORTS
• Portfolios, developmental transcripts, resumes
• Reports of student service
• RESEARCH EVALUATION DATA
• Needs assessments self-studies

39
Quantitative Data

• Summaries of objective responses on a
  questionnaire or program evaluation
• Statistics about use by students and various
  population subgroups
• Needs assessments, follow-up studies, and
  self-study reports
• Institutional research reports and fact books

40
Quantitative Data Examples

• Financial resource statements
• Assessment reports
• Research, assessment, and evaluation data staff
  activity reports
• Reports of scholarship or other professional
  contributionsStudent activity reports
• Employer reports
• Program evaluations
• Participant evaluations
• Needs assessment
• Follow-up studies
• Program evaluations

41
Qualitative Data

• Focus group information
• Written summaries of responses to open-ended
  questions in interviews and on evaluations
• Client satisfaction surveys, self-reports, and
  written comments, both solicited and unsolicited

42
Qualitative Data Examples

• Student recruitment materials
• Brochures and other sources of information about
  the programParticipation policies/proceduresPart
  icipant evaluationsProgram documentsMission
  statements Catalogs and related statementsStaff
  and student manualsPolicies and procedure
  statementsEvaluation and periodic
  reportContractsStaff memosInstitutional
  administrative documentsStatements about program
  purpose and philosophy relative to other
  educational programsOrganization chartsStudent
  and staff profilesFollow-up studiesProgram
  evaluationsPreviously published institutional
  self-study reportsVitaeAnnual reports of
  performanceService to other departmentsPortfolio
  sDevelopmental transcriptsOther evidence of
  student contributions to the institution,
  community, or professionReports of special
  student accomplishments exit interviewStudent
  journalsStudent diariesStudent
  papersObservations

43
3 Compile Review the Evidence (cont.)

• SAGs offer a ready format for evaluation
• In most instances, there are multiple criterion
  statements for each standard
• Each criterion measure focuses on a particular
  aspect of the standard, allowing raters to
  express more detailed and specific judgments

44
Criterion Measure Rating Scale

• ND 1 2 3 4 NR
• Not Done Not met Minimally Well
  Fully Not Rated
• Met Met Met

ND 1 2 3 4 NR Not Done Not met
Minimally Well Fully Not
Rated Met Met Met

• Using this scale, consider each criterion
  statement and decide
• The extent to which each criterion measure has
  been met by the program or service

45
Assessment Criteria Example

• Part 6 FINANCIAL RESOURCES
• Counseling Services (CS) must have adequate
  funding to accomplish its mission and goals.
  Funding priorities must be determined within the
  context of the stated mission, goals, objectives,
  and comprehensive analysis of the needs and
  capabilities of students and the availability of
  internal or external resources.
• CS must demonstrate fiscal responsibility and
  cost effectiveness consistent with institutional
  protocols.
• ND 1 2 3
  4 NR
• Not Done Not Met Minimally Met Well
  Met Fully Met Not Rated
• PART 6. FINANCIAL RESOURCES (Criterion
  Measures) Rating Scale NOTES
• 6.1 The program has adequate funding to
  accomplish its mission and goals. ND 1 2 3
  4 NR
• 6.2 Funding priorities are determined within
  the context of program mission,
• student needs, and available fiscal
  resources. ND 1 2 3 4 NR
• 6.3 The program demonstrates fiscal
  responsibility and cost effectiveness
• consistent with institutional protocols. ND
  1 2 3 4 NR
• Part 6 Financial Resources Overview Questions

46
4 Judge Performance

• For each of the 13 Parts, identify the criterion
  measure item number(s) for which there is
  substantial rating discrepancy.
• Items not circled should reflect consensus among
  raters that practice in that area is
  satisfactory.
• Items where judgment variance occurs need to be
  discussed thoroughly by team members.
• Follow this action by determining which practices
  can be designated as excellent or
  unsatisfactory.
• List the items requiring follow-up action,
  including any criterion measure rated as
  unsatisfactory by any reviewer.
• Complete the Overview Questions at the end of
  each Part.

47
4 Judge Performance 5 Write the Action
Plan

• Work Form A Assessment, Ratings, and
  Significant Items
• Work Form B Follow-Up Actions
• Work Form C Summary Action Plan

48

Action Plans

• Actions required for the program to meet all
  standards
• Areas that need follow-up because they are less
  than satisfactory
• Resources that would be necessary for program
  enhancements
• Dates for completion
• Staff member responsible for completing work

49
Timeline

• It is Time Consuming
• Requires about 4 5 Months to Complete
• Up to a Year
• Dont Rush It
• Better Student Learning is the Goal

50
QUIZ

• The CAS Knowledge Game
• (SAG CD-ROM)

51
Assessment Tips

• Add learning outcomes to program evaluations
• Add assessment expectations to position
  descriptions
• Ask questions about assessment experience when
  interviewing new staff

52
Tips (contd)

• Give reading assignments when training and offer
  pre-tests and post-tests
• Quick, lean, and over the phone or in combination
  with on-line surveys
• If using qualitative methodologies, be sure you
  are knowledgeable and trained

53
Tips (contd)

• Consider longitudinal studies
• Do not be afraid to communicate results and when
  you do offer executive summaries and access to
  the full report
• Consider electronic portfolios and automated data
  collectors
• Compare against national data

54
Tips (contd)

• Search for students who could serve as
  inexpensive consultants
• Have access to SPSS windows
• Systematize the process, for example include
  survey to a mailing you already distribute

55
Tips (contd)

• Do not confuse assessments with personnel
  evaluations, however, include willingness to
  assess
• Do not engage in assessment if you are not
  willing to adapt to what the data indicates

56
Roadblocks to Assessment

• Discussion
• Proactive Reactive Solutions

57
Resources

• www.cas.edu
• SAG CD PowerPoint Presentation E-Learning
  Course
• Christina Anastasi
• canastasi_at_netscape.net
• [phone]