Assessing instructional and assessment practice: What makes a lesson formative? CRESST conference, September 2004 UCLA Sunset Village, CA

1
Assessing instructional and assessment
practiceWhat makes a lesson formative?CRESST
conference, September 2004UCLA Sunset Village, CA

• Dylan Wiliam
• ETS
• dwiliam_at_ets.org

www.dylanwiliam.net
2
What makes a lesson formative?

• Moving from I know it when I see it
• Expanding ideas of formative assessment
• Feedback
• Elicitation and feedback
• The students role
• Systems thinking

3
Preliminary assumptions

• The primary purpose of educational research is
  the improvement of education.
• The purpose of education is the improvement of
  student achievement
• The improvement of student achievement will be
  achieved primarily through changes in what
  happens in classrooms
• The role of the teacher is not to teach but to
  create situations in which students learn

4
Functions of assessment

• For evaluating institutions
• For describing individuals
• For supporting learning
• Monitoring learning
• Whether learning is taking place
• Diagnosing (informing) learning
• What is not being learnt
• Forming learning
• What to do about it

5
Effects of feedback

• Kluger DeNisi (1996)
• Review of 3000 research reports
• Excluding those
• without adequate controls
• with poor design
• with fewer than 10 participants
• where performance was not measured
• without details of effect sizes
• left 131 reports, 607 effect sizes, involving
  12652 individuals
• Average effect size 0.4, but
• Effect sizes very variable
• 40 of effect sizes were negative

6
Effects of formative assessment

• Several major reviews of the research
• Natriello (1987)
• Crooks (1988)
• Black Wiliam (1998)
• Nyquist (2003)
• All find consistent, substantial effects

7
Kinds of feedback (Nyquist, 2003)

• Weaker feedback only
• Knowledge or results (KoR)
• Feedback only
• KoR clear goals or knowledge of correct results
  (KCR)
• Weak formative assessment
• KCR explanation (KCRe)
• Moderate formative assessment
• (KCRe) specific actions for gap reduction
• Strong formative assessment
• (KCRe) activity

8
Effects of formative assessment (HE)
No. Effect
Weaker feedback only 31 0.16
Feedback only 48 0.23
Weaker formative assessment 49 0.30
Moderate formative assessment 41 0.33
Strong formative assessment 16 0.51
9
Formative assessment

• Classroom assessment is not (necessarily)
  formative assessment
• Formative assessment is not (necessarily)
  classroom assessment

10
Formative assessment
Assessment for learning is any assessment for
which the first priority in its design and
practice is to serve the purpose of promoting
pupils learning. It thus differs from assessment
designed primarily to serve the purposes of
accountability, or of ranking, or of certifying
competence. An assessment activity can help
learning if it provides information to be used as
feedback, by teachers, and by their pupils, in
assessing themselves and each other, to modify
the teaching and learning activities in which
they are engaged. Such assessment becomes
formative assessment when the evidence is
actually used to adapt the teaching work to meet
learning needs. Black et al, 2002
11
Bad items/good items 1

• Which is larger 0.33 or 1/3?
• 0.33 is larger than 1/3
• 1/3 is larger than 0.33
• They are the same
• You need more information to be sure

12
Bad items/good items 2

• What is the general rule for the following
• pattern 3, 7, 11, 15, ?
• 3  n
• n  4
• 3n  4
• 4n - 1
• 4n  3

13
Bad items/good items 3

• If f  g 8, then f  g  h  ?
• 9
• 12
• 15
• 16
• 8  h

14
Bad items/good items 4

• Which of the following statements are true ?
• A square has two 90 angles
• A square is a trapezoid
• A kites are squares
• A trapezoid is a rectangle
• A rectangle is a trapezoid

15
Bad items/good items 5

• Ice-cubes are added to a glass of water. What
  happens to the level of the water as the
  ice-cubes dissolve?
• A) The level of the water drops
• B) The level of the water stays the same
• C) The level of the water increases
• D) You need more information to be sure

16
Bad items/good items 6

• What is the pH of gasoline?
• A) Less than zero
• B) Zero
• C) Greater than zero but less than 7
• D) 7
• E) Greater than 7
• F) None of the above

17
Bad items/good items 7

• In what year did World War II begin?
• A) 1919
• B) 1937
• C) 1938
• D) 1939
• E) 1941
• F) 1942

18
Good formative items are different

• Can have more (or less!) than one correct answer
• Items need to be generative
• of learning
• of insights into learning
• of insights into how to promote learning
• Increasing ability may reduce your chance of
  getting the item correct
• Distractors must be explicitly connected to
  incorrect or incomplete conceptions (facets)
• Item responses must provide clues to effective
  action
• Some items support both summative and formative
  functions well most do not

19
How do we make good items?

• Good items are not universal
• In some domains, facets of thinking are
  reasonably universal (e.g. physics)
• In most domains, facets of thinking are dependent
  on instruction
• So there is no universal set of facets
• And we have to rely on teachers to do it for
  themselves
• Good items have to be used
• Requires a model of effective, scalable teacher
  professional development

20
Why research hasnt changed teaching

• The nature of expertise in teaching
• Aristotles main intellectual virtues
• Episteme knowledge of universal truths
• Techne ability to make things
• Phronesis practical wisdom
• What works is not the right question
• Everything works somewhere
• Nothing works everywhere
• Whats interesting is under what conditions
  does this work?
• Teaching is mainly a matter of phronesis, not
  episteme

21
Countdown
3
25
1
4
9
Target number 127
22
Aspects of formative assessment
Where the learner is Where they are going How to get there
Teacher Evoking information Curriculum philosophy Feedback
Peer Peer-assessment Sharing criteria Peer-tutoring
Learner Self-assessment Sharing criteria Self-directed learning
23
A formative lesson

• Grade 9 Algebra lesson
• Weather balloon set off, rising at 8fs-1
• 2s later, a flare is launched, initial velocity
  80fs-1
• Students complete a table and a graph to answer
  three questions
• Maximum height of flare
• How long to max height
• Average velocity of flare

24
Regulation of learning

• Teaching as engineering learning environments
• Key feature well-regulated
• Teaching vs. learning
• Regulation of activity vs. regulation of learning
• Long feedback cycles vs. variable feedback cycles
• Quality control vs. quality assurance in learning

25
Regulation of learning

• Proactive (upstream) regulation
• Planning regulation into the learning environment
• Planning for evoking information
• Interactive (downstream) regulation
• Negotiating the swiftly-flowing river
• Moments of contingency
• Tightness of regulation (goals vs horizons)
• Retrospective regulation
• Structured reflection (e.g. lesson study)

26
(No Transcript)