Developing Plans to Assess Student Learning Palmer College of Chiropractic Susan Hatfield March 2005 [e-mail]

1
Developing Plans to Assess Student
LearningPalmer College ofChiropracticSusan
HatfieldMarch 2005SHatfield_at_winona.edu
2
Outline

• Basic Understandings about Assessment
• Developing Program Level Assessment Plans
• Implementing Assessment Plans
• Mistakes to avoid

3
Common Reactions to Assessment Initiatives

• Ignoring it
• Bribing someone else to do it
• Complaining about it
• Losing sleep over it
• Sitting down and writing it

4
Common Reactions to Assessment Initiatives

• Ignoring it - denial
• Bribing someone else to do it - bargaining
• Complaining about it - anger
• Losing sleep over it - depression
• Sitting down and writing it - acceptance

5
Stages of Death and Dying

• Ignoring it - denial
• Bribing someone else to do it - bargaining
• Complaining about it - anger
• Losing sleep over it - depression
• Sitting down and writing it - acceptance

6
Assessment

• Assessment is an ongoing process aimed at
  understanding and improving student learning. It
  involves
• making our expectations explicit and public
• setting appropriate criteria and high standards
  for learning quality
• systematically gathering, analyzing, and
  interpreting evidence to determine how well
  performance matches those expectations and
  standards
• using the resulting information to document,
  explain, and improve performance.

7
Assessment

• Assessment is an ongoing process aimed at
  understanding and improving student learning. It
  involves
• making our expectations explicit and public
• setting appropriate criteria and high standards
  for learning quality
• systematically gathering, analyzing, and
  interpreting evidence to determine how well
  performance matches those expectations and
  standards
• using the resulting information to document,
  explain, and improve performance.

8
Assessment

• When it is embedded effectively within larger
  institutional systems, assessment can help us
  focus our collective attention, examine our
  assumptions, and create a shared academic culture
  dedicated to assuring and improving the quality
  of higher education (Thomas A. Angelo, AAHE
  Bulletin, November 1995, p.7).

9
Assessment

• When it is embedded effectively within larger
  institutional systems, assessment can help us
  focus our collective attention, examine our
  assumptions, and create a shared academic culture
  dedicated to assuring and improving the quality
  of higher education (Thomas A. Angelo, AAHE
  Bulletin, November 1995, p.7).

10
Basic Understandings
11

• 1. Assessment initiatives evolve

12
Evolutionary Trajectories
1
2
4
5
6
7
8
3
MATURING
MAKINGPROGRESS
BEGINNING
13
Evolutionary Trajectories
1
2
4
5
6
7
8
3
MATURING
MAKINGPROGRESS
BEGINNING
14
Evolutionary Trajectories
1
2
4
5
6
7
8
3
MATURING
MAKINGPROGRESS
BEGINNING
15
Evolutionary Trajectories
1
2
4
5
6
7
8
3
MATURING
MAKINGPROGRESS
BEGINNING
16
Evolutionary Trajectories
1
2
4
5
6
7
8
3
MATURING
MAKINGPROGRESS
BEGINNING
17
Evolutionary Trajectories
1
2
4
5
6
7
8
3
MATURING
MAKINGPROGRESS
BEGINNING
18
Evolutionary Trajectories
1
2
4
5
6
7
8
3
MATURING
MAKINGPROGRESS
BEGINNING
19
2. Language of Assessment
20
Language of Assessment

• A. General skill or knowledge category

21
Language of Assessment

• A. General skill or knowledge category
• GOAL

22
Language of Assessment

• A. General skill or knowledge category
• GOAL
• B. Specific accomplishments to be achieved

23
Language of Assessment

• A. General skill or knowledge category
• GOAL
• B. Specific accomplishments to be achieved
  OUTCOME

24
Language of Assessment

• A. General skill or knowledge category
• GOAL
• B. Specific accomplishments to be achieved
  OUTCOME
• C. Activities and Assignments to help students
  learn

25
Language of Assessment

• A. General skill or knowledge category
• GOAL
• B. Specific accomplishments to be achieved
  OUTCOME
• C. Activities and Assignments to help students
  learn LEARNING EVENTS

26
Language of Assessment

• A. General skill or knowledge category
• GOAL
• B. Specific accomplishments to be achieved
  OUTCOME
• C. Activities and Assignments to help students
  learn LEARNING EVENTS
• D. The key elements related to the accomplishment
  of the outcome

27
Language of Assessment

• A. General skill or knowledge category
• GOAL
• B. Specific accomplishments to be achieved
  OUTCOME
• C. Activities and Assignments to help students
  learn LEARNING EVENTS
• D. The key elements related to the accomplishment
  of the outcome COMPONENTS

28
Language of Assessment

• E. The objects of analysis

29
Language of Assessment

• E. The objects of analysis OBJECTS

30
Language of Assessment

• E. The objects of analysis OBJECTS
• F. Data indicating degree of achievement

31
Language of Assessment

• E. The objects of analysis OBJECTS
• F. Data indicating degree of achievement
  CHARACTERISTICS

32
Language of Assessment

• E. The objects of analysis OBJECTS
• F. Data indicating degree of achievement
  CHARACTERISTICS
• G. Combination of data indicating relative
  degree of achievement of the learning outcome

33
Language of Assessment

• E. The objects of analysis OBJECTS
• F. Data indicating degree of achievement
  CHARACTERISTICS
• G. Combination of data indicating relative
  degree of achievement of the learning outcome
  INDICATORS

34
Goals
35
Goals

• Organizing Principle
• Category or Topic Area
• Subjects

36
Learning Outcomes
37
Student Learning Outcomes
Goal
38
Student Learning Outcomes
Goal
Outcome
Outcome
Outcome
Outcome
Outcome
39
Learning Events
40
Learning Events

• Assignments (in class and out of class)
• Feedback on practice
• Self evaluation
• Peer evaluation
• Role Play
• Pre Tests
• Simulation

41
Learning Objects
42
Student Learning Outcomes
Goal
Outcome
Outcome
Outcome
Outcome
Outcome
43
Student Learning Outcomes
Goal
Outcome
Outcome
Outcome
Outcome
Outcome
Learning events
Object
44
Components
45
Student Learning Outcomes
Goal
Outcome
Outcome
Outcome
Outcome
Outcome
Learning events
Object
46
Student Learning Outcomes
Goal
Outcome
Outcome
Outcome
Outcome
Outcome
component
Object
Evaluative elements
component
component
47
Performance Characteristics
48
Student Learning Outcomes
Goal
Outcome
Outcome
Outcome
Outcome
Outcome
component
Object
Evaluative elements
component
component
49
Student Learning Outcomes
Goal
Outcome
Outcome
Outcome
Outcome
Outcome
Evaluative criteria
component
Object
component
component
Characteristics
50
Indicators
51
Student Learning Outcomes
GOAL
Outcome
Outcome
Outcome
Outcome
Outcome
Degree to which outcome is achieved
component
Object
component
indicator
component
component
52
3. Assessment Plans
53
Mission
GOAL
GOAL
GOAL
GOAL
54
Mission
GOAL
GOAL
GOAL
GOAL
outcome
outcome
outcome
outcome
55
Mission
GOAL
GOAL
GOAL
GOAL
outcome
outcome
outcome
outcome
component
component
component
component
component
component
component
component
component
component
component
component
56
4. How Assessment Works
57
How Assessment Works
Year 2
Year 3
Year 1
O U T C O M E
Compare Against Benchmarks, Standards, Past
Performance
New / Revised Activity 1 New / Revised Activity
2 New / Revised Activity 3
New / Revised Activity 1 New / Revised Activity
2 New / Revised Activity 3
Activity 1 Activity 2 Activity 3
Measure Measure Measure Measure BASELINE
Measure Measure Measure Measure
Measure Measure Measure Measure
58
5. Difference between Program Effectiveness and
Outcomes Assessment
59
Assessment of Program Effectiveness

• What the program will do or achieve
• Curriculum
• Retention
• Graduation
• Placement
• Satisfaction (graduate and employer)

60
Assessment of Student Learning Outcomes

• What students will do or achieve
• Direct Methods of Assessment

61
6. Assessment Requires Exertion and Intention
62
Exertion without Intention
63
Exhaustion
64
Intention without Exertion
65
Intention and Exertion
66

• 7. Measure what you value

67

• 7. Measure what you value then value what you
  measure

68
Developing Plans toAssess Student Learning
69
Assessment Plans
70
Developing an Assessment Plan

• 1. Agree on your Mission

71
Mission

• Where are we located and with whom are we
  affiliated?
• Who do we serve?
• How do we serve?
• What results are anticipated?
• What is our philosophy of operation?
• What are the special and unique features of this
  department / unit / program?
• What are our future plans?

72
Mission

• When was the mission last reviewed?
• How does the mission of the program fit with the
  mission of the institution?
• Where is the mission published?

73
Developing an Assessment Plan

• 2. Identify your program goals
• major theories
• major theorists
• professional / technical skills
• research methodologies
• communication
• ethics
• professional development

74
Mission
75
Mission
GOAL
GOAL
GOAL
GOAL
76
Mission
Clinical Performance
Problem Solving
Communication
Professional Development
77
Mission
Diagnosis
Treatment
Patient care
Business Management
78
Developing an Assessment Plan

• 3. Identify your program level student learning
  outcomes

79
Mission
Diagnosis
Treatment
Communication
Professionalism
outcome
outcome
outcome
outcome
80
Mission
Diagnosis
Treatment
Communication
Professionalism
outcome
outcome
outcome
81
Mission
Diagnosis
Treatment
Communication
Professionalism
outcome
outcome
outcome
82
Mission
Diagnosis
Treatment
Communication
Professionalism
outcome
outcome
outcome
83
Student Learning Outcomes

• Students should be able to
• ltltaction verbgtgt ltltsomethinggtgt

84
Student Learning Outcomes

• Learner Centered
• Specific
• Action oriented
• Cognitively Appropriate

85
COMPREHENSION
EVALUATION
APPLICATION
SYNTHESIS
KNOWLEDGE
ANALYSIS
Associate Classify Compare Compute Contrast Differ
entiate Discuss Distinguish Estimate Explain Expre
ss Extrapolate Interpolate Locate Predict Report R
estate Review Tell Translate
Analyze Appraise Calculate Categorize Classify Com
pare Debate Diagram Differentiate Distinguish Exam
ine Experiment Identify Inspect Inventory Question
Separate Summarize Test
Arrange Assemble Collect Compose Construct Create
Design Formulate Integrate Manage Organize Plan Pr
epare Prescribe ProducePropose Specify Synthesize
Write
Appraise Assess Choose Compare Criticize Determine
Estimate Evaluate Grade Judge Measure Rank Rate R
ecommend Revise Score Select Standardize Test Vali
date
Cite Count Define Draw Identify List Name Point Qu
ote Read Recite Record Repeat Select State Tabulat
e Tell Trace Underline
Apply Calculate Classify Demonstrate Determine Dra
matize Employ Examine Illustrate Interpret Locate
Operate Order Practice Report Restructure Schedule
Sketch Solve Translate Use Write
86
Possible Outcomes

• Students will appreciate.
• Students will be exposed to.
• Students will demonstrate knowledge of the
  history, theories and applications of
• Students will attend.
• Faculty will provide students with opportunities
  to.

87
Possible Outcomes

• Students will appreciate
• Students will be exposed to.
• Students will demonstrate knowledge of the
  history, theories and applications of
• Students will attend.
• Faculty will provide students with opportunities
  to.

88
Program Level Student Learning Outcomes

• Design
• Critique
• Plan
• Analyze
• Direct
• Demonstrate
• Execute

89
Developing an Assessment Plan

• 4. Identify the assignments and activities that
  demonstrate achievement of each learning outcome

90
Learning Objects

• There are multiple objects that can demonstrate
  student learning
• Authentic or created specifically for purposes of
  assessment

91
Learning Objects

• Standardized Exam, Abstract, advertisement,
  annotated bibliography, biography, briefing,
  brochure, budget, care plan, case analysis,
  chart, cognitive map, court brief, debate,
  definition, description, diagram, dialogue,
  diary, essay, executive summary, exam, flow
  chart, group discussion, instruction manual,
  inventory, lab notes, letter to the editor,
  matching test, mathematical problem, memo, micro
  theme, multiple choice test, narrative, news
  story, notes, oral report, outline, performance
  review, plan, precis, presentation, process
  analysis, proposal, regulation, research
  proposal, review of literature, taxonomy,
  technical report, term paper, thesis, word
  problem, work of art. (Walvoord Anderson 1998).

92
Learning Objects

• Standardized Exam, Abstract, advertisement,
  annotated bibliography, biography, briefing,
  brochure, budget, care plan, case analysis,
  chart, cognitive map, court brief, debate,
  definition, description, diagram, dialogue,
  diary, essay, executive summary, exam, flow
  chart, group discussion, instruction manual,
  inventory, lab notes, letter to the editor,
  matching test, mathematical problem, memo, micro
  theme, multiple choice test, narrative, news
  story, notes, oral report, outline, performance
  review, plan, precis, presentation, process
  analysis, proposal, regulation, research
  proposal, review of literature, taxonomy,
  technical report, term paper, thesis, word
  problem, work of art. (Walvoord Anderson 1998).

93
Developing an Assessment Plan

• 5. Identify the components of successful
  achievement of the outcome
• -What are we looking for to be able to tell
  whether or not student performance is acceptable?

94
Components

• Key evaluative elements
• Outcome specific, not object specific

95
Components
Goal
Outcome
Outcome
Outcome
Outcome
Outcome
96
Components
Goal
Outcome
Outcome
Outcome
Outcome
Outcome
component
Evaluative elements
component
component
97
Components
Goal
Outcome
Outcome
Outcome
Outcome
Outcome
component
component
component
98
Components
Goal
Outcome
Outcome
Outcome
Outcome
Outcome
component
component
component
99
Components
Goal
Outcome
Outcome
Outcome
Outcome
Outcome
component
Object
component
component
100
Possible Components

• Comprehension
• Problem Identification
• Organization
• Acceptance of Responsibility
• Initiative / Motivation
• Creativity
• Task Completion
• Collaboration
• Participation
• Attitude
• Responsiveness

101
Possible Components

• Technical Factors (mass - kVp relationship)
• Positioning
• Tube / receptor
• Film identification
• Exposure Factors
• Shielding
• Beam limiting devices
• Repeat rates
• Patient instructions

102
Possible Components

• Language - instructions
• Language - explanation
• Immobilization
• Age or situation specific positioning
• Technical factors
• Knowledge of pathology
• Infection control
• Knowledge of trauma views

103
Developing an Assessment Plan

• 6. Identify characteristics of effective /
  accurate / successful performance for each
  Component / Trait

104
Performance Characteristics

• Scale or description for assessing each of the
  components /traits
• Two to Five-point scales for each component /
  trait

105
Performance Rubric
Performance Characteristics
Exceeds Expectations
Meets Expectations
Does not meet Expectations
Components
Descriptions of Unacceptable, Acceptable, or
Excellent Performance
106
Performance Rubric
Performance Characteristics
Exceeds Expectations
Meets Expectations
Does not meet Expectations
Components
107
Goal
Outcome
Component
Component
Component
Component
108
Clinical Competence
Produce radiographic images of diagnostic quality
Technical Factors mAs
Positioning
Tube / Image receptor align
Film ID
109
Performance Characteristics

• Level or degree
• Accurate, Correct
• Depth, Detail
• Coherence, Flow
• Complete, Thorough
• Integration
• Creative, Inventive
• Evidence based, supported
• Engaging, enhancing

110
Performance Characteristics

• Missing - Included
• Inappropriate - Appropriate
• Incomplete - Complete
• Incorrect - Partially Correct - Correct
• Vague - Emergent - Clear
• Marginal - Acceptable - Exemplary
• Distracting - Neutral - Enhancing
• Usual - Unexpected - Imaginative
• Ordinary - Interesting - Challenging

111
Performance Characteristics

• Simple - More fully developed - Complex
• Reports - Interprets - Analyzes
• Basic - Expected - Advanced
• Few - Some - Several - Many
• Isolated - Related - Connected - Integrated
• Less than satisfactory - satisfactory - more than
  satisfactory - outstanding
• Never - Infrequently - Usually - Always

112
Communication
Speak in public situations
Verbal Delivery
Several - Some - Few fluency problems
Nonverbal Delivery
Distracting - Enhancing
Structure
Disconnected - Connected - Integrated
Evidence
Doesnt support - Sometimes - Always supports
113
MIDTERM EXAM
114
Learning Outcome Rules

• Use one cognitive level
• Focus on outcomes, not processes (what, not how)
• List single accomplishments
• Do not indicate level of quality (effective)
• Allow for multiple learning objects

115
Example 1

• Gather factual information and apply it to a
  given problem in a manner that is relevant,
  clear, comprehensive, and conscious of possible
  bias in the information selected
• BETTER Students will be able to apply factual
  information to a problem
• COMPONENTS
• Relevance
• Clarity
• Comprehensiveness
• Aware of Bias

116
Example 1

• Gather factual information and apply it to a
  given problem in a manner that is relevant,
  clear, comprehensive, and conscious of possible
  bias in the information selected
• BETTER Students will be able to apply factual
  information to a problem
• COMPONENTS
• Relevance
• Clarity
• Comprehensiveness
• Aware of Bias

117
Example 1

• Gather factual information and apply it to a
  given problem in a manner that is relevant,
  clear, comprehensive, and conscious of possible
  bias in the information selected
• BETTER Students will be able to apply factual
  information to a problem
• COMPONENTS
• Relevance
• Clarity
• Comprehensiveness
• Aware of Bias

118
Example 2

• Imagine and seek out a variety of possible goals,
  assumptions, interpretations, or perspectives
  which can give alternative meanings or solutions
  to given situations or problems
• BETTER Students will be able to provide
  alternative solutions to situations or problems
• COMPONENTS
• Variety of assumptions, perspectives,
  interpretations
• Analysis of comparative advantage

119
Example 2

• Imagine and seek out a variety of possible goals,
  assumptions, interpretations, or perspectives
  which can give alternative meanings or solutions
  to given situations or problems
• BETTER Students will be able to provide
  alternative solutions to situations or problems
• COMPONENTS
• Variety of assumptions, perspectives,
  interpretations
• Analysis of comparative advantage

120
Example 2

• Imagine and seek out a variety of possible goals,
  assumptions, interpretations, or perspectives
  which can give alternative meanings or solutions
  to given situations or problems
• BETTER Students will be able to provide
  alternative solutions to situations or problems
• COMPONENTS
• Variety of assumptions, perspectives,
  interpretations
• Analysis of comparative advantage

121
Example 3

• Formulate and test hypotheses by performing
  laboratory, simulation, or field experiments in
  at least two of the natural science disciplines
  (one of these experimental components should
  develop, in greater depth, students laboratory
  experience in the collection of data, its
  statistical and graphical analysis, and an
  appreciation of its sources of error and
  uncertainty)
• BETTER Students will be able to test
  hypotheses.
• COMPONENTS
• Data collection
• Statistical Analysis
• Graphical Analysis
• Identification of sources of error

122
Example 3

• Formulate and test hypotheses by performing
  laboratory, simulation, or field experiments in
  at least two of the natural science disciplines
  (one of these experimental components should
  develop, in greater depth, students laboratory
  experience in the collection of data, its
  statistical and graphical analysis, and an
  appreciation of its sources of error and
  uncertainty)
• BETTER Students will be able to test
  hypotheses.
• COMPONENTS
• Data collection
• Statistical Analysis
• Graphical Analysis
• Identification of sources of error

123
Example 3

• Formulate and test hypotheses by performing
  laboratory, simulation, or field experiments in
  at least two of the natural science disciplines
  (one of these experimental components should
  develop, in greater depth, students laboratory
  experience in the collection of data, its
  statistical and graphical analysis, and an
  appreciation of its sources of error and
  uncertainty)
• BETTER Students will be able to test
  hypotheses.
• COMPONENTS
• Data collection
• Statistical Analysis
• Graphical Analysis
• Identification of sources of error

124
ImplementingAssessmentPlans
125
Implementing Assessment Plans
126
Implementing Assessment Plans

• 1. Identify which core courses or elective
  course cluster support which program level
  learning outcomes
• Required Classes
• Course 1
• Course 2
• Course 3
• Course 4
• Select one course from Course 5
• Course 6
• Course 7
• Course 8

127
Student Learning Outcomes
Course 1
Course 2
Course 3
Course 4
Cluster 1
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
128
Implementing Assessment Plans

• 2. Check for Orphan Outcomes and Empty
  Requirements

129
Student Learning Outcomes
Course 1
Course 2
Course 3
Course 4
Cluster 1
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
130
Student Learning Outcomes
Course 1
Course 2
Course 3
Course 4
Cluster 1
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
131
Implementing Assessment Plans

• 2. Check for Orphan Outcomes
• 3. Identify the level at which each course
  supports the learning outcome
• Knowledge / Comprehension
• Application / Analysis
• Synthesis / Evaluation

132
Student Learning Outcomes
Course 1
Course 2
Course 3
Course 4
Cluster 1
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
133
Student Learning Outcomes
Course 1
Course 2
Course 3
Course 4
Cluster 1
K
A
S
K
A
S
A
A
S
K
A
S
K
A
S
K
A
S
K
A
S
134
Implementing Assessment Plans

• 4. Identify Assessment Points in the curriculum

135
Student Learning Outcomes
Course 1
Course 2
Course 3
Course 4
Cluster 1
K
A
S
K
A
S
A
A
S
K
A
S
K
A
S
K
A
S
K
A
S
136
Student Learning Outcomes
Course 1
Course 2
Course 3
Course 4
Cluster 1
K
A
S
K
A
S
A
A
S
K
A
S
K
A
S
K
A
S
K
A
S
137
Implementing Assessment Plans

• 5. Phase in implementation of assessment of
  student learning outcomes

138
Phase One
Student Learning Outcomes
Course 1
Course 2
Course 3
Course 4
Cluster 1
K
A
S
K
A
S
A
A
S
K
A
S
K
A
S
K
A
S
K
A
S
139
Phase Two
Student Learning Outcomes
Course 1
Course 2
Course 3
Course 4
Cluster 1
K
A
S
K
A
S
A
A
S
K
A
S
K
A
S
K
A
S
K
A
S
140
Phase Three
Student Learning Outcomes
Course 1
Course 2
Course 3
Course 4
Cluster 1
K
A
S
K
A
S
A
A
S
K
A
S
K
A
S
K
A
S
K
A
S
141
Phase Four
Student Learning Outcomes
Course 1
Course 2
Course 3
Course 4
Cluster 1
K
A
S
K
A
S
A
A
S
K
A
S
K
A
S
K
A
S
K
A
S
142
Implementing Assessment Plans

• 6. Aggregate data

143
Goal Student Learning Outcome
Course level report
Does not meet
Meets
Exceeds
Component
13
65
17
7
Component
21
72
54
Component
14
32
144
Goal Student Learning Outcome
PROGRAM WIDE COMPETENCY REPORT
Does not meet
Meets
Exceeds
Component
20
65
15
57
30
Component
13
58
Component
24
18
145
Big MistakesIn Assessment
146
Big Mistakes in Assessment

• Assuming that it will go away
• Trying to do too much, too soon
• Expecting to get it right the first time
• Not considering implementation issues when
  creating plans

147
Big Mistakes in Assessment

• Borrowing plans and methods without acculturation
• Demanding statistical research standards
• Doing it for accreditation instead of improvement

148
Big Mistakes in Assessment

• Confusing institutional effectiveness with
  student learning
• Making assessment the responsibility of one
  individual
• Assuming collecting data is Doing Assessment