CS-90 Strategies for Aligning Assessment for Program Improvement with Accountability for Individual Student Learning Thomas P. Pusateri, Ph.D. Florida Atlantic University [e-mail] Concurrent session 110th Annual Meeting of the Commission on

1
CS-90Strategies for Aligning Assessment for
Program Improvement with Accountability for
Individual Student LearningThomas P. Pusateri,
Ph.D.Florida Atlantic Universitypusateri_at_fau.edu
Concurrent session110th Annual Meeting of the
Commission on CollegesSouthern Association of
Colleges and SchoolsAtlanta Hyatt Regency
HotelDecember 5, 2005, 345-445 PM
2
http//iea.fau.edu
3
(No Transcript)
4
(No Transcript)
5
Background Information

• Increased emphasis on identifying and assessing
  core student learning outcomes in higher
  education (e.g., Measuring Up 2004).
• The Florida Board of Governors included student
  achievement as 1 of 7 of its accountability
  measures.
• A Board of Governors resolution (April 22, 2004)
  directed university personnel to develop Academic
  Learning Compacts (ALCs) and related processes.

6
What are ALCs?

• University personnel will provide students
  and prospective students with Academic Learning
  Compacts, which include concise statements of
  what active and successful participants in the
  joint teaching-learning process will know and be
  able to do, expressed in terms of the core
  student learning outcomes embodied in the
  requirements for each baccalaureate degree. The
  Compacts also will list the types of assessments
  used in the program. Compacts provided to
  students must be written in a user-friendly,
  jargon-free format.
• Policy Guideline 04.08.26
• State University System (of Florida)
• Division of Colleges and Universities

Florida Atlantic Universitys is realigning its
assessment system to accommodate Academic
Learning Compacts.
7
BoG Policy Guideline FAUs Realignment

• University personnel will develop Academic
  Learning Compacts that
• I.a. Identify, at a minimum, the expected
  core student learning outcomes for
  program graduates in the areas of
• content/discipline knowledge and skills
• communication skills
• critical thinking skills
• these skills may be defined at either the
  university level or the program level

• FAUs programs identify one or more of each of
  the following outcomes
• Content knowledge
• Vocabulary, Theories
• Research skills
• Technical skills
• Communication skillsWrittenOralOtherTeam
• Critical Thinking skillsAnalytical Creative
  Practical
• FAUs Writing Across the Curriculum requirement

8
BoG Policy Guideline FAUs Realignment

• I.b. Identify corresponding assessments used to
  determine how well student learning matches
  those articulated expectations.
• Where/When are students assessed?
• How are students held to performance standards?

• Departments assess all students through one or
  more of the following
• Core required courses
• Capstone experience
• Senior thesis
• Senior project/portfolio
• Field placement
• Standardized test

9
BoG Policy Guideline FAUs Realignment

• Departments establish standards through one or
  more approaches
• Standardizing courses
• Common syllabi
• Common expectations
• Common assignments
• Grading rubrics
• Faculty define levels of student performance
• Periodic calibration
• Setting cut-off scores on required exams

I.b. Identify corresponding assessments used to
determine how well student learning matches
those articulated expectations. Where/When are
students assessed? How are students held to
performance standards?
10
BoG Policy Guideline FAUs Realignment

1. University personnel will develop the evaluation
   systems (including external validations)
   necessary to corroborate that the assessments
   referenced above measure student achievement
   against the expected core learning outcomes and
   results will be used to improve student
   achievement and program effectiveness.

• FAUs departments rely on one or more systems
• Multiple raters
• Juried performances
• 2 faculty members
• Non-faculty raters
• SamplingCollection and periodic review of
  student work
• Comparisons to norms on standardized tests

11
BoG Policy Guideline FAUs Realignment

• University personnel shall develop an action
  plan that will be approved by their universitys
  board of trustees and submitted to the Division
  of Colleges and Universities. The plan shall
  include
• a. a proposed timeline for developing policies
  and implementing procedures to capture each
  element outlined above, as well as in making
  Academic Learning Compacts readily available to
  students.

• FAUs action plan
• Briefings for Board of Trustees and University
  Undergraduate Programs Committee (Fall 04)
• Consultations with academic departments (04-05)
• Progress report to Faculty Senate Chair (Mar 05)
• Board of Trustees - Workshop (Apr 05)- Approves
  timeline (May 05)
• Faculty Senate approves ALC Guidelines (Oct 05)
• Board of Trustees approves ALC Policies (Nov 05)
• ALCs will be made available to students (Dec 05)

12
(No Transcript)
13
(No Transcript)
14
BoG Policy Guideline FAUs Realignment
REVIEW OF ALCs University personnel are expected
to demonstrate how information from the periodic
review of student learning outcomes, as well as
from the evaluation of corresponding assessment
mechanisms, has been used to improve student
achievement and program effectiveness.
Departments will report their evaluation
(assessment) strategies by February 1, 2006.
15
Reporting samples from core required courses or
capstones on the distribution of scores obtained
on embedded questions, key assignments, rubrics,
etc.
16
Assessing multiple raters
17
Assessing multiple raters
18
FAUS DEFINITIONS OF CONTENT KNOWLEDGE IN THE
DISCIPLINE Declarative knowledge Students will
demonstrate knowledge of the vocabulary, history,
theories or concepts specific to the discipline.
Declarative knowledge is usually assessed via
in-class or standardized tests, typically in an
objective (multiple-choice short answer) format.
Procedural knowledge Research skills
Students will demonstrate knowledge of the
procedures involved in discipline-specific
research (e.g., idea generation, literature
review, data collection, reporting). These
skills are usually measured via the evaluation of
a research project, thesis, or dissertation. Proce
dural knowledge Technical skills Student will
demonstrate technical skills related to the
discipline (e.g., developing a business report or
lesson plan, mastering a musical instrument,
learning how to use statistical software). This
type of procedural knowledge may be assessed via
an in-class project (e.g., case study, paper),
observation (e.g., juried performance), or
portfolio.
19
FAUS DEFINITIONS OF COMMUNICATION SKILLS Written
communication Students will produce writing
that is grammatically correct, well-organized,
and properly formatted for the purpose of the
assignment and the discipline. This is usually
assessed via individual paper assignments where
instructors assess the quality of written
expression. Oral communication Students will
prepare and deliver informative and/or persuasive
oral presentations that attend to the audience
and are well-organized. This is usually assessed
via individual oral presentations in a classroom
setting, although sometimes instructors and/or
supervisors may assess oral communication across
time (e.g., during classroom discussions or at an
internship site). Other forms of communication
Students will prepare and present information
and/or persuasive material using media that may
or may not include written or oral communication
(e.g., a musical or theatrical performance a
visual, audiovisual, or graphical product). This
is usually assessed via classroom projects,
juried performances, and/or portfolios. Team/Colla
borative communication Students will
demonstrate team-oriented, collaborative skills
in which they contribute to group products. This
is usually assessed via group papers or
presentations and may involve peer-assessment
(e.g., group members assess each other) and/or
assessment by audience members (e.g., an
instructor, supervisor, or peer).
20
BoG Policy Guideline FAUs Realignment

• b. A description of how university personnel will
  certify that each baccalaureate graduate has
  completed a program with clearly articulated core
  student learning expectations and that
  corresponding robust and effective assessment
  mechanisms have been used to ensure that
  graduates have met the criteria of the Compacts.

• FAU will certify students through the awarding of
  a baccalaureate degree from programs that have
  articulated ALCs and that require students to
  achieve acceptable performance standards prior to
  graduation via completion of one or more of the
  following (a) senior capstone course, (b)
  exit examination(c) a series of core
  required courses.

21
FAUS DEFINITIONS OF CRITICAL THINKING
SKILLS Analytical skills Students will
analyze, evaluate, compare/contrast or judge
discipline-specific theories, issues, events, or
other content. This is usually assessed through
examining the quality of argument in a student
essay, oral presentation, or formal report.
Creative skills Students will create a product
by synthesizing knowledge from a discipline
(e.g., create a new piece of art, interpret a
piece of music in a personal and appropriate way,
develop a new theory or research proposal). This
is usually assessed by an instructor or
supervisor examining a student product for the
quality of its synthesis of current knowledge
into a new product. Practical skills Students
will put into practice their knowledge and skills
within a discipline (e.g., developing a business
proposal or lesson plan, writing a grant). This
is assessed ideally by examining an authentic
product the student employs in an appropriate
setting (e.g., a student teacher delivering a
lesson, an intern proposing a plan to a
supervisor) it may also be assessed via in-class
projects in which students propose practical
solutions based on their understanding of the
discipline (e.g., case study analysis). Based on
Robert Sternbergs Theory of Successful
Intelligence
22
EXAMPLES OF ALCs CONTENT
KNOWLEDGE PSYCHOLOGY (Terms, theories)
Graduates in Psychology will understand basic
concepts, theories, and experimental findings in
four core areas of psychology (cognitive,
developmental, social, and psychobiology).
HISTORY (Research Technical skills)
Students will demonstrate the ability to conduct
archival research utilizing both primary and
secondary sources. They will develop procedural
and technical skills involved in historical
research such as library research, oral history,
book review writing, and proper citations.
FINANCE (Technical skills) Students will
develop proficiency in the use of Microsoft Excel
to solve financial problems. 
23
EXAMPLES OF ALCs COMMUNICATION
SKILLS GEOGRAPHY Graduates in Geography will be
able to produce writing that is grammatically
correct and well-organized and to deliver clear
and well-organized oral presentations. Both
written and oral communications should illustrate
clear communication skills utilizing graphic
tools in geographic information systems, remote
sensing and other techniques. REAL ESTATE
Students will demonstrate their abilities to
report the results from their feasibility
analyses both orally and in written form in a
clear and grammatically sound manner. Students
will also demonstrate skills in discussing the
analyses presented by their peers. TEACHER
EDUCATION Students will demonstrate the ability
to communicate effectively verbally and in
writing (EAP 2, Indicator 2.2) by successfully
developing and implementing a lesson plan.
24
EXAMPLES OF ALCs CRITICAL THINKING
SKILLS ENGLISH Students will analyze/interpret
a complex literary work, demonstrating the
capacity to formulate and develop in a formal
essay a significant topic in depth and in an
independent and even original way. Essays are
expected to exhibit mature creative and
analytical thinking, and an ability to create a
mature critical argument quickly with no external
assistance. COMPUTER ANIMATION IN ARTS Students
will demonstrate proficiency in understanding and
apply the stages of problem-solving skills (e.g.
research, thumbnail, roughs, comprehensive and
mechanical preparation) as applied to 3D computer
arts. ELECTRICAL ENGINEERING Design and
conduct scientific and engineering experiments
including analysis and interpretation of data.
Deliver engineering results that meet performance
standards for cost, safety, and quality. Make and
defend ethical judgments in keeping with
professional standards.
25
CORE REQUIRED COURSES PSYCHOLOGY For each of
the courses listed below, faculty teaching that
course will develop a master syllabus which will
include student learning outcomes to be addressed
in all sections of the course. Faculty will share
strategies for developing course assignments
(e.g., exams, exercises, term papers) to assess
student attainment of the learning outcomes. The
courses areEXP 3505 Cognition,DEP 3054
Psychology of Human Development,SOP 3004
Social Psychology, and PSB 3002 Biological
Bases of Behavior. MUSIC All students must
complete four specific courses in the history and
literature of music MUH 3013 (Introduction to
Music History), MUH 4211 (Music of Western
Civilization I), MUH 4212 (Music of Western
Civilization II), and MUH 4371 (20th Century
Music History).
26
CAPSTONE EXPERIENCE SOCIAL WORK All BSW
students are required to take SOW 4510 (Field
Education in Social Work), which includes a
classroom field seminar plus 400 agency-based
hours supervised by a professional social worker.
This experience integrates classroom knowledge
from the Human Behavior in the Social Environment
(HBSE) sequence, practice sequence, research
sequence, and policy sequence students use this
knowledge base to develop skills to intervene
with various clients. HONORS COLLEGE During
the senior year, students will enroll in thesis
research and thesis writing and will produce a
draft honors thesis to be evaluated. To prepare
for writing a senior thesis, students will
complete (1) courses in their concentration aimed
at providing the content knowledge needed to
write a thesis, (2) writing courses aimed at
providing competence in effective written
communication, and (3) courses in the core and
concentration designed to build critical thinking
skills.
27
STANDARDIZED TEST NURSING Approximately three
months prior to graduation, all seniors will take
the Health Education Systems, Inc. (HESI)
examination. This test is a valid predictor of
performance on the National Licensing Examination
(NCLEX). Students must achieve a score of 850 or
better on the HESI to graduate. Students who do
not receive a passing score are provided with
advising and a remediation plan and the
opportunity to re-take the exam they will not
graduate until their score is 850 or better.
OCEAN ENGINEERING One of the requirements for
graduation is that a student must take the
nationally-administered Fundamentals of
Engineering (FE) examination. The examination
tests one's knowledge of basic and fundamental
engineering subjects statics, dynamics,
thermodynamics, fluid mechanics, computer
programming, engineering economics, electrical
networks and strength of materials.
28
STANDARDIZING COURSES PSYCHOLOGY Faculty
teaching core courses will develop a master
syllabus which will include student learning
outcomes to be addressed in all sections of the
course. Faculty will share strategies for
developing course assignments (e.g., exams,
exercises, term papers) to assess student
attainment of the learning outcomes. ACCOUNTING
Faculty members teaching the undergraduate
accounting core courses ACG 3131 (Financial
Reporting), ACG 3341 (Managerial Decision Making
and Accounting)., and TAX 4001 (Federal Taxation
I) are required to teach a set number of chapters
from a common text selected by departmental
decision. Faculty members teaching a particular
course will develop and periodically review a
common list of educational objectives and the
topics and critical thinking skills appropriate
to these objectives to teach in each course.
29
GRADING RUBRICS ENGLISH Essays in all 3000- and
4000-level English courses will be assessed
holistically according to a rubric developed and
shared by faculty and communicated to students.
SUPERIOR Addresses the question or prompt fully
Demonstrates substantial comprehension of
relevant material Shows substantial depth,
complexity, and creativity of thought
Demonstrates clear and coherent organization
Develops arguments fully with ample supporting
detail Demonstrates superior control of diction,
syntactic variety, and transitions
STRONGAddresses the question or prompt
substantively, yet not fully Demonstrates
comprehension of relevant material Shows depth,
complexity, and creativity of thought
Demonstrates clear organization Develops
arguments with supporting detail Demonstrates
control of diction, syntactic variety, and
transition
30
ENGLISH RUBRIC (continued) COMPETENT Adequately
addresses the question or prompt Demonstrates
adequate understanding of relevant material
Shows clarity of thought but may treat the topic
simplistically or repetitively Demonstrates
adequate organization Develops arguments
adequately, with some detail Demonstrates
adequate facility with syntax, mechanics, and
usage but contains some errors INADEQUATE May
distort or neglect parts of the question or
prompt, and/or Fails to comprehend relevant
material Lacks clarity of thought demonstrates
confused/simplistic thinking, and/orLacks
adequate organization, and/or Fails to provide
adequate or appropriate details to support
generalizations, or may provide details without
generalization, and/or Demonstrates significant
errors in language, syntax, or mechanics
31
HONORS COLLEGE RUBRIC Honors with distinction
The thesis demonstrates a profound knowledge of
the fundamental concepts in the subject area.
The thesis shows mastery of grammar, syntax,
structure and style it is lucid, well-organized,
and stylistically elegant. The thesis engages in
sophisticated critical thinking and rigorously
considers alternate resolutions of a problem in
an exemplary fashion or otherwise shows
sophisticated critical thinking in an exemplary
manner appropriate to the area of study. Honors
The thesis shows a thorough knowledge of the
fundamental concepts in the subject area. The
thesis shows effective command of grammar,
syntax, structure and style it is clearly
written and sensibly organized. The thesis
engages in sophisticated critical thinking and
explores alternate approaches or resolutions of a
problem effectively or otherwise shows
sophisticated critical thinking in a manner
appropriate to the area of study. Unacceptable
The thesis does not show competent knowledge of
the fundamental concepts in the subject area.
The thesis does not show mastery of grammar,
syntax, structure, or style. The thesis does not
competently engage in critical thinking or
explore alternate approaches or resolutions of a
problem.
32
CUT-OFF SCORE ON STANDARDIZED EXAM NURSING
Approximately three months prior to graduation,
all seniors will take the Health Education
Systems, Inc. (HESI) examination. This test is a
valid predictor of performance on the National
Licensing Examination (NCLEX). Students must
achieve a score of 850 or better on the HESI to
graduate. Students who do not receive a passing
score are provided with advising and a
remediation plan and the opportunity to re-take
the exam they will not graduate until their
score is 850 or better. COMPARISON TO NORMS ON
STANDARDIZED TESTS
33
MULTIPLE RATERS MUSICCONTENT KNOWLEDGE
(Technical skills) andCOMMUNICATION andCRITICAL
THINKING (Creative skills) Graduates will have
ability to perform a cross section of the music
from the complete repertoire of the performance
medium and exhibit the appropriate technical
skills for artistic self-expression. Students
in the Bachelors Music program will be evaluated
by the appropriate faculty at jury performances
each semester. They are also required to
successfully complete a senior recital, which
must include representative examples from the
appropriate repertoire for the performance
medium. A pre-hearing evaluated by three faculty
members must be approved before the recital is
scheduled.
34
SAMPLING LANGUAGES LINGUISTICS Will randomly
sample the performance of students writing term
papers for program-specific 3000- and 4000-level
courses to compare demonstrated research skills
for these with the research-skill performance of
students taking and passing FOL 3880. Will
randomly sample the accuracy of linguistic
structures and attention to linguistic nuance in
the essays of students enrolled in selected 3000-
and 4000-level courses to compare rankings of
student performance in linguistics-specific
courses with rankings of students linguistic
awareness as evidenced in other courses. JEWISH
STUDIES Periodically selected exams and papers
will be read by members of the Jewish Studies
Executive Committee to ensure that the criteria
of the Academic Learning Compacts are being met
by individual students.
35
National Center for Public Policy and Higher
Education
Measuring Up
2004 report
INCOMPLETE PLUS
36
Florida Board of Governors Accountability Measures

1. Number/Percent of students from underserved
   populations enrolled/graduating
2. 4- and 6-year graduation rates
3. Access to and production of degrees
4. Meeting statewide workforce needs
5. Research expenditures from external sources
6. World-class academic research programs
7. Student learning Academic Learning Compacts

37
(No Transcript)
38
(No Transcript)