Anchoring Assessment in its Place: Reviving, Surviving, and Even Thriving

1
Anchoring Assessment in its PlaceReviving,
Surviving, and Even Thriving

• Douglas Eder
• deder_at_siue.edu

Southern Illinois University Edwardsville
2
Strongest LinksThe Scholarship of Teaching

• Clear Goals
• Adequate Preparation
• Appropriate Methods
• Significant Results
• Effective Presentation
• Reflective Critique

3
Asteroid Impact AssessmentAssumptions
Expectations?
4
Assessment vs. Evaluation

• Assessment focuses on the student and the
  learning environment.
• Evaluation focuses on the professor and the
  teaching performance.

5
Direct vs. Indirect Assessment

• Direct assessment acquires evidence about student
  learning and the learning environment Exams,
  projects, logs, portfolios, observations....
• Indirect assessment acquires evidence about how
  students feel about learning and their learning
  environment Surveys, questionnaires, interviews,
  focus groups, reflective essays....

6
We Give Grades, Dont We?

• Arent grades
• (by themselves)
• enough?

NO!
7
Grades may reflect many things besides student
mastery of course objectives

• Verbal ability
• Participation
• Cooperation
• Extra credit
• Attendance
• Effort
• Criterion Performance vs. Value Added
• Myths regarding student evaluations

8
When used alone, grades are no longer regarded as
adequate indicators of student learning.

• Professor autonomy Grades in one course or
  section may be recorded using a different
  standard than grades in another.
• General accountability Much of the current
  public annoyance with higher education comes from
  a lack of skill and content mastery by students
  who have received good grades.

9
Why Assessment Wont Go Away

• Limits on money
• Shift in the public view of Higher Education
• Result We must demonstrate quality of outcome
  and return on investment.
• Through assessment educators meet
  responsibilities to themselves, to students, and
  to the public.

10
Cognitive Neuroscience

• The more of the brain --sensory, motor, and
  association pathways-- involved during learning,
  the more effective the learning process.

11

• Deep (expert) learning is a curriculum function.
• It is not a course function.

Universities are systems
12
Assessment Asks Good Questions

• Bad questions take just as much energy to answer
  as good questions.

13
The Basic Assessment Question

• What do we want to know about ourselves?

14
The Basic Assessment Question Amplified

• Are we doing what we say were doing?
• Are we doing what we ought to be doing?
• Just what do we think were doing?

15
Focus on the Students

• Its not so much what the professors do that
  matters.
• Its what the students do.
• . . . . .
• So, how do we know what students do?

16
Good Assessment Asks Good Questions

• What is your job description for your first year
  students? For your second year students? How do
  they differ?
• Do we have a curriculum or a set of courses?
  Which of these do our students experience?
• How much and what kinds of writing do our
  students do? What kinds of math? Are these
  relevant to our curricular goals?

17
Directed Paraphrase

• In 1-3 clear, concise sentences, please write
  your definition of deep learning (in a discipline
  or in general education). Construct a definition
  that would make sense to your students regardless
  of their years in school or academic disciplines.

18
Outline

• Clear goals -- we identify the question
• Adequate preparation -- we see the task
• Appropriate methods -- we can do the task
• Significant results -- we do care
• Effective presentation -- we can escape
• Reflective critique -- we can improve

19
Principle 1

• Clear
• Goals

20
Whats the quickest way to solve a maze?
21
What do you think are the most desirable
characteristics for new employees to have?

• Self motivation -- training is expensive
• Business etiquette -- ability to represent
• Written communication -- can explain a complex
  topic simply
• Oral communication -- group and individual
• Disciplinary competence is assumed

22
What Are Your Goals for Your Students?

• What are your Big Rocks for your students? What
  do you want your students to look like when they
  walk out the door after spending time under your
  supervision?

23
What do you think your new graduates should look
like when they walk out your door?

• Goal
• Goal
• Goal
• Goal
• Goal

24
Student goals are not necessarily what the
faculty pictures for them.
25
The mission of the Black Walnut Inn is

• Firstly
• That guests experience accommodations and
  hospitality that meet the highest standards
  expected by the discerning traveler, and that
  this will encourage the novice Bed and Breakfast
  guests to repeat their choice.
• Secondly
• That the owners experience a relaxed lifestyle
  that includes friendly relationships with their
  guests, a workload that retains enthusiasm, and a
  monetary return that avoids financial concerns.

26
The Importance of Goals

• Cheshire Puss..., asked Alice, would you tell
  me, please, which way I ought to go from here?
• That depends a good deal on where you want to
  get to, said the Cat.
• I don't much care where said Alice.
• Then it doesn't matter which way you go, said
  the Cat.

27
Approximate Analogy

• INSTRUCTIONS Complete the analogy below.
• 1. Assessment is to teaching as
• ______________ is to _______________.
• Explain __________________________________
• __________________________________
• __________________________________

28
Approximate Analogy

• INSTRUCTIONS Complete the analogy below.
• 1. F2F learning is to e-learning as
• _______________ is to _______________.
• Explain __________________________________
• __________________________________
• __________________________________

29
Its human nature to respond to subtle cues in
the environment
30
The Importance of Goals

• If you dont watch
• where youre going,
• youre liable to wind up somewhere else.
• --attributed to Yogi Berra

31
Principle 2

• Adequate
• Preparation

32
Thinking About Learning

• What do we know about how mammals learn?

They must see the bar. The bar must be
steady. There must be a fish.
33
A Good Rule for Practicing Medicine and
Everything Else

• Diagnose first Prescribe second

34
Four Barriers to Institutional Improvement

• Amnesia
• Fantasia
• Inertia
• Nostalgia
• --after Shulman

35
Why do we insist on

• measuring with a micrometer?
• when we mark it with chalk?
• and cut it with an axe?
• ----after Peter Ewell

36
SIUE Assessment Mantra

• Assessment monitors student learning
• It does not evaluate faculty teaching.

37
The Basic Assessment Question

• What do we want to know about ourselves?

38
Angelos Rule

• Caveat Assessor
• If you dont want to know, dont ask.

39
How Assessment Causes Change

• What is measured is valued
• What is ignored doesnt exist.

40
Goals and Objectivesfor Students

• Goals express what we want our students to be.
• Objectives describe what we want our students to
  do.
• .....

• Objectives are indicators of goals.

41
Objectives to be assessed...

• ...must be written so that they specify visible
  student behaviors or products.

42
An Assessment Question How Do You Know...

• ...that students walk out your door looking like
  you want them to? What behaviors have they
  exhibited or products have they produced? What
  are the indicators for your goals?

43
Principle 3

• Método Apropiado

44
Whatever Assessment Is...

• ...Its Multiple Measures Over Time.

45

• If all you have is a hammer, everything starts to
  look like a nail.

46
Some Assessment Ways and Means

• Assessment days or centers
• Case studies
• Classroom assessments
• Completion and retention studies
• Content analyses
• Debates
• Direct observations
• Focus groups
• Graduate success
• Internships and service learning
• Interviews (including videotapes)
• Exams for certification and licensure

• Matrices
• Performances
• Portfolios of several kinds
• Projects (Primary Trait Analysis)
• Questionnaires and surveys (Direct and
  telephone employer, alumni, and student attitude
  and satisfaction)
• Reflective essays
• Study and activity logs
• Tests
• (Locally-developed and standardized)
• Transcript analyses

47
An Effective Consequence of the Basic Assessment
Question

• Assess the things that matter within your
  university culture.
• --Pat Hutchings

48
Course or Curriculum Alignment Grid

• What are your students focusing on?

49
Curriculum AlignmentAn Alternative View
50
Assessment Autopista

• A. What are you already doing and what
  opportunities could you pursue in order to do
  assessment?

What Ways and Means (p.11) might you use?
51
Cautions and Obstacles toAssessment Initiatives
52
Assessment Barrera

• B. What resource do you need?
• C. What or who is standing in the way of using
  assessment to monitor student learning?

53
Principle 4

• Significant
• Results

54
The Main Thing

• is to keep the Main Thing

the Main Thing.
55
How do you know that your students can do there
what they do here?
56
The Flight Simulator...

• ...combines teaching, learning, and assessing
  into a real task.
• Because assessment is embedded in teaching and
  learning, it is largely transparent.

57
The Academic Simulator...

• ...is a reflective, scholarly engagement between
  student and dedicated professor that results in a
  visible product or behavior.

58
Authentic Assessment is...

• appraisal of a real task
• in a real or simulated environment
• under supervision

The closer any simulation is to the real thing,
the better the appraisal.
59
Traits of the Senior Assignment (SRA)(SIUEs
Academic Simulator)

• Embedded in learning and teaching
• Matches departmental goals
• Departmentally owned
• Viewed by the faculty
• High stakes assessment
• Liberal education component

60
High Stakes Assessment
61
STUDENT LEARNING IS AUTHENTICALLY ASSESSED IN
PUBLIC ENVIRONMENTS

• Civil Engineering Solid waste reduction, campus
  seismic analyses, campus water run-off, water
  treatment, dam repair
• Chemistry Student presentations and defenses
  before external referees, including Nobel
  Laureate Roald Hoffmann
• Psychology Presentations and defenses at annual
  meeting of Midwest Psychology Association in
  Chicago
• Art Design Mexica95, Mexica96, ... Mexica02
  exhibitions
• Speech Communication Academic/Student Affairs
  connections by studying academic advising,
  minority scholarship use, conflict resolution
  resident halls, University Speech Center
• Liberal Studies American Sign Language teaching
  protocol for college lectures
• Nursing Community health outreach in Mexico City

62
Direct Curriculum Feedbackfrom Senior Assignment
Findings

• Chemistry Added writing and speaking at junior
  level increased sophomore writing emphasis.
• Psychology Increased statistics, hypothesis
  forming and testing shifted teaching emphasis
  toward research design.
• Biology Added student presentations upgraded
  instruction in computer use, speech, graphics,
  and statistics.
• Mathematics Statistics Increased writing and
  computer use emphasized mathematical
  experimentation.
• Music Increased collaborative learning by
  emphasizing ensemble performance opportunities.
• Business Increased analytical writing, critical
  thinking, ethical understanding, working in teams.

63
Primary Trait Analysis...

• ...is a very strong link.

64
What Were Really Looking for is

• Way Better than Good Enough
• Good Enough
• Not Good Enough
• -------------------------------

That's Enough!
65
The Faculty Sets the Standards
66
What is Good Effective Critical Thinking?

• Identifying
• Exploring
• Prioritizing
• Revisioning
• --Wolcott Lynch

67
Critical Thinking in a Speaking
EnvironmentGrading and Assessing

• Traits 3 2 1
• Critical Thinking (after Wolcott Lynch)
• 1. Identifying __ __ __
• 2. Exploring __ __ __
• 3. Prioritizing __ __ __
• 4. Revisioning __ __ __
• Speaking (after Martell)
• 5. Voice and pace __ __ __
• 6. Body language, mannerisms __ __ __
• 7. Professionalism, audience awareness __
  __ __
• 8. Use of media __ __ __
• 9. Handling of questions __ __ __
• Scoring
• 27-23 Exceeds expectations
• 22-16 Meets expectations
• 15- 9 Does not meet expectations

68
Critical Thinking in a Speaking
Environment Levels of Performance
Exploring 2 3 Probes alternatives and presents
primary and secondary evidence in support. 2
Recognizes alternatives and acknowledges
existence of evidence in support. 1 Does not
recognize that alternatives may exist ignores
conflicting evidence.
Prioritizing 3 3 2 1
Use of media 8 3 2 1
69
Critical Thinking in a Speaking
EnvironmentGrading and Assessing

• Traits 3 2 1
• Critical Thinking (after Wolcott Lynch)
• 1. Identifying _x_ ___ ___
• 2. Exploring ___ _x_ ___
• 3. Prioritizing ___ _x_ ___
• 4. Revisioning ___ ___ _x_
• Speaking (after Martell)
• 5. Voice and pace _x_ ___ ___
• 6. Body language, mannerisms ___ ___ _x_
• 7. Professionalism, audience awareness ___
  _x_ ___
• 8. Use of media ___ _x_ ___
• 9. Handling of questions _x_ ___ ___
• Score 19
• Scoring
• 27-23 Exceeds expectations
• 22-16 Meets expectations
• 15- 9 Does not meet expectations

70
Critical Thinking in a Speaking
EnvironmentGrading and Assessing

• Traits 3 2 1
• Critical Thinking (after Wolcott Lynch)
• 1. Identifying 12 13 5
• 2. Exploring 11 15 4
• 3. Prioritizing 8 16 6
• 4. Revisioning 3 12 15
• Speaking (after Martell)
• 5. Voice and pace 22 5 3
• 6. Body language, mannerisms 14 12 4
• 7. Professionalism, audience awareness 16
  9 5
• 8. Use of media 4 15 11
• 9. Handling of questions 6 16 8
• Mean score 19.2 or 2.1 / 3 ? 2.8 / 4
• QUESTION What part(s) of the curriculum
• deserves special attention?

71
Assessment is...

• ...more than finding out how many students passed
  and what the average score is on a test of
  competence.

72
Choose one objective and...

• ...write a 3- or 4- point standard or rubric that
  represents either qualitative or quantitative
  assessment of learning.

73
The Minute Paper

• 1. What is the most important thing you learned
  in this session?
• 2. What is your most important unanswered
  question from our time together so far?

74
Principle 5

• Effective
• Presentation

75
The Minute Paper

• 1. What is the most important thing you learned
  in this session?
• 2. What is your most important unanswered
  question from our time together so far?

76
Institutional Effectiveness

• A comparison of results achieved to goals
  intended.
• --Ewell, 1985

77
A Picture of Accountability
78
What To Assess?

• 1 2 3
• ? Incoming ? Curriculum ?Teaching Learning
• Student Processes
• ?
• 5 4
• ? Outgoing Student ? Outcome
• --after Vars

79
The Three Curricula

• The one in the catalog
• The one the professors teach
• The one the students experience

80
Three Parts of a Public Assessment Report

• What have you learned?
• What are you going to do about it?
• When and how are you going to do it?

81
Administrivia Requirement

• One hour

Per department
Per year
82
Assessment Reports

• Who writes them?
• Who reads them?
• What happens to them?

83
Attracting the Audience Is Important
84
Principle 6

• Middle States, AACSB, Baldrige, NCATE, ABET, and
  others all require
• Reflective
• Critique

85
An Important Lesson from the Farm

• A pig doesnt get any fatter merely by weighing
  it.

86
What Next?
87
What is the weakest link?

• Missing or inappropriate feedback
• Little or no faculty involvement
• Does not address goals of curriculum
• Strictly norm based

88
Guiding Principles for AssessmentAngelo, 1997

• Actively engages participants
• Addresses what matters most
• Operates at key points and times
• Embeds in normal activities
• Focuses on processes as well as products
• Provides timely, relevant feedback to those who
  need it

89
Closing Thought

• The enemy of the good is the perfect.

90
Assessment vs. Evaluation

• Assessment focuses on the student and the
  learning environment.
• Evaluation focuses on the professor and the
  teaching performance.

91
Examples of Things Students are Qualified to
Evaluate

• Perceived fairness
• The professor tested over what I was expected
  to learn.
• Communication effectiveness
• The professor communicated in language and with
  examples that I could understand.
• Perceived challenge
• I felt stretched by the challenges of this
  course.
• Availability
• The professor was available to answer my
  questions during office hours or through
  e- communication.
• Timeliness
• The professor began and ended class on time.

92
Examples of Things Students are Not Qualified to
Evaluate

• Professors preparation
• The professor was prepared for class.
• Currency of information
• The professor was up-to-date in his/her field.
• Relevance
• The content of this course is relevant to the
  discipline.

93
Some Areas of Ambiguity

• Expectations
• This class was what I expected.
• Degree of difficulty
• This class challenged me at an appropriate
  level.
• Organization
• The professor was organized.
• Engagement/interactivity
• This course engaged me to learn well.

94
End-of-semester evaluations are one way for
professors to receive formative feedback and
correct errors.
95
Dangerous Question 9

• Compared with other instructors and other courses
  at this level, I rate this instructor/course
  overall as _____

96
Myths?

• Higher ratings go to easier graders.
• Profs who teach larger classes get lower ratings.
• Rating scales influence ratings
• Ratings are stable

• Students learn more from highly rated professors.
• Student ratings and trained observer rating
  agree.
• Students rate style and enthusiasm.
• Student ratings truly reflect instructional
  quality.

97
EvaluationsSummative and Formative
98
EvaluationsSummative and Formative
99
Asking students what they liked or valued
about their classes...was far less productive
than asking them what they had specifically
gained.--Linda Nilson, Putting Real Value in
Student Evaluations, POD presentation
100
Slides by Friday

• También, web recurso
• http//www.siue.edu/
• deder/assess/index.html
• with links to major assessment sites worldwide

101
Regla de 15/35

• In any organization
• 15 leaders, opinion shapers
• (Includes dreamers)
• 35 willing followers, early adopters
• --------------------------------------------------
  --
• 35 reluctant followers, late adopters
• 15 curmudgeons

102
15 Curmudgeons

• 5 already busy and productive
• Leave them alone
• 5 reasoned skeptics
• Take them to lunch
• 5 just plain mean
• Leave them alone

103
Don Farmers Advice

• I know this bothers you and you dont want to
  think about it. But...

104
...please dont prevent us from thinking about
it.
105
To get faculty buy-in...

• Start small.
• Use pilot projects to capture the 15 leaders
  35 early adopters.

106
Fertilize...

• ...then get out of the garden.

107
We have never been asked...

• ...to assess everything all at once.

108
7 Principles of Good Practice in Effective
Undergraduate Education

• Encourages contact between students and faculty
• Develops reciprocity and cooperation among
  students
• Encourages active learning
• Gives prompt feedback on performance
• Emphasizes time on task
• Communicates high expectations
• Respects diverse talents and ways of learning

109
My Goals

• Facilitate a conversation about education goals
• Provide some assessment techniques
• Outline assessment organization for
  accountability
• Give some materials for you to keep
• Give some time to think
• Give some time to share
• Enjoy

110
The Minute Paper

• 1. What is the most important thing you learned
  in this session?
• 2. What is your most important unanswered
  question from our time together so far?

111
Most Important Thing

• Assessment is not the same thing as evaluation of
  faculty 5
• Goals and objectives are different 5
• Assessment has realistic steps and guidelines 3
• The focus of program assessment is on student
  learning 3
• Assessment does not have to be complicated 2

112
Important Unanswered Question

• How do we operationalize assessment? 2
• I need more help to identify indicators 2
• How do I do assessment in a large class? 1
• Teaching vs. learning -- when I assess learning I
  find out much about my teaching 1
• I have questions about qualitative assessment
  because of terms like adequate and
  appropriate 1
• I need self confidence 1

113
Important Unanswered Question

• How do we apply specific techniques? 1
• How to apply assessment techniques to
  administrative departments 1
• Is this all possible? 1
• How does one write an assessment report? 1
• How to use assessment for improvement? 1
• What are valid program objectives?
• Assessment may take away the most important
  educational goals 1

114
Blooms Taxonomy

• LEVEL SOME COGNITIVE BEHAVIORS
• Evaluation Appraisal of an Analysis
• or Synthesis
• Synthesis Assembly of Application
• Analysis Disassembly of Application
• Application Use of Understanding
• Understanding Management of Knowledge
• Knowledge Memorization of facts, language,
  concepts, principles, theories

115
Analytical Minute PaperAnalysis via Blooms
TaxonomySIUE 2003 NFO
116
Analytical Minute PaperAnalysis via Blooms
Taxonomy -- UPR
117
Annotated Word Journal

• Read the assigned text and write one word that
  captures the essence of what youve read and
  summarizes your response to it.
• ____________________
• Explain why you chose that word and how it
  provides, in a capsule, your summary of the
  reading.
• ______________________________
• _______________________________

118
Assessing Group Effectiveness

• 1. What specific goal(s) is this group trying to
  accomplish? Please list the goals(s) in your
  priority order. Do you think the group basically
  agrees on the contents of this list?
• 2. What activities has the group specifically
  chosen to undertake or assign in order to achieve
  its goals? Which activities, if any, are
  particularly effective?
• 3. Does each group member have specific --even
  unique-- responsibilities that help the group
  attain its goal(s)? List all group members by
  name and their individual responsibilities.

119
Assessing Group Effectiveness

• 4. Do you find the work of your group stimulating
  and worth your time? How many hours per week do
  you spend working with this group? In the table
  below, enter the percentage of these hours spent
  in each category of effectiveness.
• 5. Does this group have the resources (e.g.,
  organization, communication, leadership, talents,
  time) to achieve its goals? What additional
  resources are needed for real effectiveness?

120
Time Log

• How do students use the 90 of their time not in
  the classroom?

121
RSQC2

• Recall
• Summarize
• Question?
• Connect
• Comment

122
Assessment of Self Confidence

• 1. Set and explain learning goals
• 2. Determine most effective pedagogy
• 3. Use the pedagogy
• 4. Motivate students to engage
• 5. Establish standards of performance
• 6. Use Primary Trait Analysis
• 7. Apply the feedback

Very Somewhat Not very Not at all Very
Somewhat Not very Not at all Very Somewhat
Not very Not at all Very Somewhat Not very
Not at all Very Somewhat Not very Not at
all Very Somewhat Not very Not at all Very
Somewhat Not very Not at all Very Somewhat
Not very Not at all Very Somewhat Not
very Not at all
123
Approximate Analogy

• INSTRUCTIONS Complete the analogy below.
• 1. Assessment is to teaching as
• _______________ is to _______________.
• Explain __________________________________
  ___________________________________
  ___________________________________

124
Select an idea about assessing student learning

• Dont filter or censor for now.
• Why do I care about this question?
• Who else cares?
• Who might benefit?
• Who might be inconvenienced?
• Upon reflection, do I want to proceed?

125
Possible Benefits

• More effective student learning?
• More efficient teaching?
• Better use of time?
• Number of people benefit?
• Improved finances?
• Potential rise in status?
• Improved cheer and good will?
• ________________________?

126
Possible Costs

• Confusion and administrivia?
• Extra work?
• Extra time?
• Inconvenienced people and committees?
• Drain on finances?
• Potential loss of status?
• Resistance and grumpiness?
• ________________________?

127
Benefit/Cost Ratio

• Reckon the benefit/cost ratio
• Benefits
• Costs
• Is the result greater or lesser than 1.0?

128
Rethink Benefits and Costs

• In order to reap potential general education
  assessment benefits, are might we give up some
  old things so that some promising new things
  might grow?
• Are there any additional questions bearing on the
  concept that this conference has raised?
• Which of these questions, if any, remain
  unresolved?

129
Dealing creatively with human obstacles
130
An Old Gaelic Blessing

• May those who love us, love us,
• And those that dont love us, may God turn their
  hearts,
• And if he doesnt turn their hearts, may he turn
  their ankles,
• So well know them by their limping.