How Do We Really Know What Our Students Are Learning Teaching and Assessing for Deep Learning

1
How Do We Really Know What Our Students Are
Learning?Teaching and Assessing for Deep Learning

• Douglas Eder
• deder_at_siue.edu

Southern Illinois University Edwardsville
2
Asteroid Impact AssessmentAssumptions
Expectations?
3
Assessment makes learning visible

• Assessment is the ongoing, systematic examination
  of student learning and the learning environment.

4
Direct vs. Indirect Assessment

• Direct assessment acquires evidence about student
  learning and the learning environment Exams,
  projects, logs, portfolios, observations....
• Indirect assessment acquires evidence about how
  students feel about learning and their learning
  environment Surveys, questionnaires, interviews,
  focus groups, reflective essays....

5
Assessment vs. Evaluation

• Assessment focuses on the student and the
  learning environment.
• Evaluation focuses on the professor and the
  teaching performance.

6
Assessment Questions Answers
7
Assessment Asks Questions

• Bad questions take just as much energy to answer
  as good questions do.

8
What is the purpose of college?
9
UTEP Goals, 2001-05Learning Teaching(President
s Page)

• Students will acquire tools to
• meet lifelong intellectual, ethical, and career
  challenges, and
• be leaders.

10
The Basic Assessment Question

• What do we want to know about ourselves?

11
The Basic Assessment Question Amplified

• Are we doing what we say were doing?
• Are we doing what we ought to be doing?
• Just what do we think were doing?

12
Good Assessment Asks Good Questions

• What is your job description for your first year
  students? For your second year students? How do
  they differ?
• Do we have a curriculum or a set of courses?
  Which of these do our students experience?
• How much and what kinds of writing do our
  students do? What kinds of math? Are these
  relevant to our curricular goals?

13
Its human nature to respond to subtle cues in
the environment
14
The Scholarship of Teaching

• Discovery
• Integration
• Application
• Teaching
• Public and peer reviewed
• Reflective

15
Six Principles of Scholarship--Glassick et al.

• 1. Clear goals -- we identify the question
• 2. Adequate preparation -- we see the task
• 3. Appropriate methods -- we can do the task
• 4. Significant results -- we do care
• 5. Effective presentation -- we can escape
• 6. Reflective critique -- we can improve

16
Principle 1

• Clear
• Goals

17
Whats the quickest way to solve a maze?
18
The First and Only Goal

• To teach for long-term retention of information
  and application to new situations
• --after Halpern Hakel

19

• Deep (expert) learning is not a course function.
• It is a curriculum function.

Colleges and Universities are Systems
20
Cognitive Neuroscience

• The more of the brain --sensory, motor, and
  association pathways-- involved during the
  learning process, the more effective is the
  learning.

21
P
s s s s s s s s s s s s s s s s s s s s s s s s s
s s s s s s s s s s s s s s s s s s s s s s s s s
s s s s s s s s s s
22
P
s s s s s s
s s s s s s
s s s s s s
s s s s s s
s s s s s s

s s s s s s
s s s s s s
23
s s s s s s s Ps s s s s s s s s
s s s s s s s s s s s s s
24
P
s s s s s s s s s s s s s
s s s s s s s
25
Why do students come to classroom?
26
Why do students come to classroom?
Why should students come to classroom?
27
Annotated Word Journal

• Read the assigned text and write one word that
  captures the essence of what youve read and
  summarizes your response to it.
• ____________________
• Explain why you chose that word and how it
  provides, in a capsule, your summary of the
  reading.
• ____________________________________
• ____________________________________

28
The Importance of Goals

• Cheshire Puss..., asked Alice, would you tell
  me, please, which way I ought to go from here?
• That depends a good deal on where you want to
  get to, said the Cat.
• I don't much care where said Alice.
• Then it doesn't matter which way you go, said
  the Cat.

29
Principle 2

• Adequate
• Preparation

30
Goals and Objectivesfor Students

• Goals express what we want our students to be.
• Objectives describe what we want our students to
  do.
• .....

• Objectives are indicators of goals.

31
To be assessment friendly, objectives should

• Focus on students
• Make the learning goals visible (serve as
  indicators)
• Describe behaviors or products (doing, making)
  that can be captured by assignments.

32
UTEP Core Curriculum Objectives?

• Communications To enable students to communicate
  effectively in clear and correct prose in a style
  appropriate to the subject, occasion, and
  audience.
• Mathematics To develop a quantitatively literate
  college graduateable to apply basic mathematical
  tools in the solution of real world problems.
• Natural Sciences To enable the student to
  understand, construct and evaluate
  relationshipsand to understand the bases for
  constructing and testing theories.
• Humanities, Visual Performing Arts To expand
  students knowledge of the human condition and
  human cultures.
• US History, Political Science To expand
  students knowledge of the origin and evolution
  of US Texas history.
• Social Behavioral Sciences To increase
  students knowledge of how social and behavioral
  scientists discover, describe, and explain
  behavior.
• Institutional Option To develop in students
  the critical thinking tools necessary to become
  effective learners.

33
An Assessment Question How Do You Know...

• ...that students walk out your door looking like
  you want them to? What behaviors have they
  exhibited or products have they produced? What
  are the indicators for your goals?

34
Principle 3

• Appropriate
• Methods

35
Some Assessment Ways and Means

• Assessment days and centers
• Case studies
• Classroom assessments
• Completion and retention studies
• Content analyses
• Debates
• Direct observations
• Focus groups
• Graduate success
• Internships and service learning
• Interviews (including videotapes)
• Exams for certification and licensure

• Matrices
• Performances
• Portfolios of several kinds
• Projects (Primary Trait Analysis)
• Questionnaires and surveys (Direct and
  telephone employer, alumni, and student attitude
  and satisfaction)
• Reflective essays
• Study and activity logs
• Tests
• (Locally-developed and standardized)
• Transcript analyses

36

• If all you have is a hammer, everything starts to
  look like a nail.

37
Course or Curriculum Alignment Grid

• What are your students focusing on?

38
The Minute Paper

• 1. What is the most important thing you learned
  here so far?
• 2. What is your most important unanswered
  question from our time together?

39
Primary Trait Analysis...

• ...is a very strong link.

40
What Were Really Looking for is

• Way Better than Good Enough
• Good Enough
• Not Good Enough
• -------------------------------

That's Enough!
41
Principle 4

• Significant
• Results

42
Five Conditions That Foster Deep
LearningHalpern Hakel, Marchese, DeWinstanley
Bjork, Shpancer,National Research Council

• Engaging students actively
• Practicing retrieval and presentation in varied
  environments
• Linking new learning to prior learning
• Applying learning to new situations that students
  care about
• Receiving timely and relevant reinforcement

43
What are the characteristics of

• first year writing --- Pink
• second year writing --- Yellow
• third year writing --- Green
• fourth year writing --- Blue

44
Cognitive Stages of Knowing(Piaget, Perry,
Kitchener, Baxter-Magolda -- Summarized by Haynes

• Absolute Transitional
• Method of authority Answers come from
  research
• Uncertainty when Uncertainty when
• authority is answers are unavailable
  unknown
• Strategy Gain control Strategy Debate and of
  information research

45
Cognitive Stages of Knowing(Piaget, Perry,
Kitchener, Baxter-Magolda -- Summarized by Haynes

• Independent Contextual
• Many valid interpretations Evidence in context
• All opinions are good, Some opinions are
  including ones own better than others
• Strategy Introspection Strategy Values are
  important

46
Cognitive Stages of Knowing(Piaget, Perry,
Kitchener, Baxter-Magolda -- Summarized by Haynes

• Absolute Transitional Independent
  Contextual
• 1st 68 32 0 0
• 2nd 47 52 1 0
• 3rd 11 83 5 1
• 4th 2 80 16 2
• 5th -- -- 57 12

47
Staged Writing Thinking...

• ...takes students from one cognitive level to the
  next through incremental, not additive,
  writing/speaking assignments.
• e.g., Description/Narration
• Analysis
• Comparison/Contrast
• Integration

48
Highest Priority Questions

• The three highest priority questions that I would
  like to look into regarding student educational
  performance at this institution are
• 1.________________________________
• 2.________________________________
• 3.________________________________

49
What is Good Effective Critical Thinking?

• Identifying
• Exploring
• Prioritizing
• Revisioning
• --Wolcott Lynch

50
Critical Thinking in a Writing EnvironmentGradin
g and Assessing

• Traits 3 2 1
• Critical Thinking (after Wolcott Lynch)
• 1. Identifying __ __ __
• 2. Exploring __ __ __
• 3. Prioritizing __ __ __
• 4. Revisioning __ __ __
• Writing
• 5. Consistent focus on topic or issue __
  __ __
• 6. Claims founded upon evidence __ __ __
• 7. Language appropriate for the audience __
  __ __
• 8. Appropriate writing mechanics __ __ __
• 9. Scholarly bibliographic support __ __ __
• Scoring
• 27-23 Exceeds expectations
• 22-16 Meets expectations
• 15- 9 Does not meet expectations

51
Critical Thinking in a Writing EnvironmentGradin
g and Assessing

• Traits 3 2 1
• Critical Thinking (after Wolcott Lynch)
• 1. Identifying __ __ __
• 2. Exploring __ __ __
• 3. Prioritizing __ __ __
• 4. Revisioning __ __ __
• Writing
• 5. Consistent focus on topic or issue __
  __ __
• 6. Claims founded upon evidence __ __ __
• 7. Language appropriate for the audience __
  __ __
• 8. Appropriate writing mechanics __ __ __
• 9. Scholarly bibliographic support __ __ __
• Scoring
• 27-23 Exceeds expectations
• 22-16 Meets expectations
• 15- 9 Does not meet expectations

52
Critical Thinking in a Writing Environment Levels
of Performance
Exploring 2 3 Probes alternatives and presents
primary and secondary evidence in support. 2
Recognizes alternatives and acknowledges
existence of evidence in support. 1 Does not
recognize that alternatives may exist ignores
conflicting evidence.
Prioritizing 3 3 2 1
Consistent focus on topic or issue 5 3 2 1
53
Critical Thinking in a Writing EnvironmentGradin
g and Assessing

• Traits 3 2 1
• Critical Thinking (after Wolcott Lynch)
• 1. Identifying _x_ ___ ___
• 2. Exploring ___ _x_ ___
• 3. Prioritizing ___ _x_ ___
• 4. Revisioning ___ ___ _x_
• Writing
• 5. Consistent focus on topic or issue _x_
  ___ ___
• 6. Claims founded upon evidence ___ ___ _x_
• 7. Language appropriate for the audience ___
  _x_ ___
• 8. Appropriate writing mechanics ___ _x_ ___
• 9. Scholarly bibliographic support _x_
  ___ ___
• Score 19
• Scoring
• 27-23 Exceeds expectations
• 22-16 Meets expectations
• 15- 9 Does not meet expectations

54
Critical Thinking in a Writing EnvironmentGradin
g and Assessing

• Traits 3 2 1
• Critical Thinking (after Wolcott Lynch)
• 1. Identifying 12 13 5
• 2. Exploring 11 15 4
• 3. Prioritizing 8 16 6
• 4. Revisioning 3 12 15
• Writing
• 5. Consistent focus on topic or issue 22 5
  3
• 6. Claims founded upon evidence 14 12 4
• 7. Language appropriate for the audience 16
  9 5
• 8. Appropriate writing mechanics 4 15 11
• 9. Scholarly bibliographic support 6 16
  8
• Mean score 19.2 or 2.1 / 3 ? 2.8 / 4
• QUESTION What part(s) of the curriculum
• deserves special attention?

55
Assessment is...

• ...more than finding out how many students passed
  and what the average score is on a test of
  competence.

56

• Construct an effective 3- or 4-point rubric for
  some aspect of writing or critical thinking.
• Pay special attention to the level identified as
  acceptable.

57
An Important Lesson from the Farm

• A pig doesnt get any fatter merely by weighing
  it.

58
Closing Thought

• The enemy of the good is the perfect.