Feedback, efficiency and learning payoff: new designs for formative assessment - PowerPoint PPT Presentation

Loading...

PPT – Feedback, efficiency and learning payoff: new designs for formative assessment PowerPoint presentation | free to view - id: f9615-ZDc1Z



Loading


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation
Title:

Feedback, efficiency and learning payoff: new designs for formative assessment

Description:

Feedback, efficiency and learning pay-off: new designs for formative assessment ... Centre for Academic Practice and Learning Enhancement (CAPLE) ... – PowerPoint PPT presentation

Number of Views:292
Avg rating:3.0/5.0
Slides: 55
Provided by: francesd3
Category:

less

Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: Feedback, efficiency and learning payoff: new designs for formative assessment


1
  • Feedback, efficiency and learning pay-off new
    designs for formative assessment
  • Professor David Nicol,
  • Deputy-Director,
  • Centre for Academic Practice and Learning
    Enhancement (CAPLE)
  • Director, REAP project (www.reap.ac.uk)
  • University of Strathclyde
  • Queens University Belfast, 15th February 2008

2
Plan
  • Describe background
  • Introduce research
  • Case studies of practice from REAP
  • Discussion
  • Thinking about your practice
  • Developments and Guidelines
  • Sources of information and advice

3
Background
  • Projects
  • Student Enhanced Learning through Effective
    Feedback (SENLEF) project funded by HE Academy
  • Literature review model of formative assessment
    and feedback 7 principles in relation to
    development of learner self-regulation
  • The Reengineering Assessment Practices (REAP)
    project funded by SFC 1m) www.reap.ac.uk
  • QAA Scotland first year experience and
    formative assessment

4
Re-engineering Assessment Practices project
  • 3 HEIs (Strathclyde, Glasgow Caledonian Business
    School, Glasgow University)
  • Large 1st year classes (160-900 students)
  • A range of disciplines (19 modules 6000
    students)
  • Many technologies online tests, simulations,
    discussion boards, e-portfolios, e-voting,
    peer/feedback software, VLE, online-offline
  • Goals learning quality and teaching efficiencies
  • Outputs case studies, advice for strategic
    change
  • Assessment for learner self-regulation

5
Why take assessment and feedback seriously?
  • A key driver of student learning
  • Major cost in higher education
  • Widely reported that students dont read feedback
    provided
  • Drop-out and retention linked to academic
    experience
  • QAA reports main area of criticism in England

6
  • Nationally only 55 of students think feedback is
    prompt and had helped to clarify things they did
    not understand Scotland 48
  • Nationally only 63 of students agree that have
    received detailed comments on their work
    Scotland 49

7
A key research paper
  • Black and Wiliam (1998) 250 studies
  • Real classroom situations tutor, peer and
    self-assessment and feedback
  • Positive benefits for learning and achievement
    across all content areas/skills and sectors
  • . the lowest effect size would move a student
    from the average into the top third in a class
    test
  • Big impact in schools but what about HE?

8
First Year The academic experience
  • What is important?
  • Coping with transition
  • Understanding what is required
  • Engagement with academic programmes
  • Support and feedback
  • Experiences of success
  • Agents in own learning
  • Belief in self (ability) and motivation
  • Social dynamics of learning (belonging)
  • Based on research by Yorke (UK) and Tinto (US)

9
Background (1)
  • Gibbs, G. Simpson, C (2004) Conditions under
    which assessment supports students learning,
    Learning and Teaching in Higher Education, 1,
    3-31.
  • See
  • Formative Assessment in Science Teaching (FAST)
    project at http//www.open.ac.uk/science/fdtl/

10
Gibbs and Simpson (2004)
  • Assessment tasks Conditions 1-4
  • Capture enough study time (in and out of class)
  • Are spread out evenly across timeline of study
  • Lead to productive activity (deep vs surface)
  • Communicate clear and high expectations
  • i.e concern here is with steers about how much
    work to do

11
Background (1)
  • Literature Review
  • Nicol, D. Macfarlane-Dick, D. (2006). Formative
    assessment and self-regulated learning A model
    and seven principles of good feedback practice.
    Studies in Higher Education, 34 (1), 199-218
  • Nicol, D Milligan, C. (2006), Rethinking
    technology-supported assessment practices in
    relation to the seven principles of good feedback
    practice. In C. Bryan K. Clegg, Innovative
    assessment in higher education, Routledge.
  • Background
  • Student Enhanced Learning through Effective
    Feedback SENLEF project funded by HE Academy
  • REAP project www.reap.ac.uk

12
Current thinking
  • Students are always engaged in self-assessment/sel
    f-regulation of their own learning (Winne, 2005
    Black Wiliam, 2005). Logically entailed by
    constructivist thinking
  • The act of using teacher feedback implies that
    self-assessment must be present
  • Feedback in HE is being reduced so how are
    students still learning
  • The question is how can we scaffold students
    learning so they become better at self-regulation
    (Lajoie, 2005)

13
Ideas in Practice
  • Consider self and peers as much as the teacher as
    sources of Assessment and Feedback
  • Tap into different qualities than teacher can
    provide
  • Saves time
  • Provides considerable learning benefits (LLL)
  • Dont focus just on written feedback but every
    step of the cycle
  • Understanding the task criteria (Sadler, 1983)
  • Applying what was learned in action

14
Scaffolding self regulation 7 principles of
good feedback (assessment design)
  • Clarify what good performance is (goals,
    criteria, standards).
  • Facilitate the development of reflection and
    self-assessment in learning
  • Deliver high quality feedback to students that
    enables them to self-correct
  • Encourage peer and student-teacher and peer
    dialogue around learning
  • Encourage positive motivational beliefs self
    esteem through assessment
  • Provide opportunities to act on feedback
  • Provide information to teachers that can be used
    to help shape their teaching
  • Source Nicol and
    Macfarlane-Dick (2006)

15
Two super principles
  • Super-principle 1 time on task and effort
    (engagement) i.e. steers on how much work to do
    and when Gibbs and Simpson 4 conditions
  • Super-principle 2 developing learner
    self-regulation (empowerment/self-regulation) i.e
    steers to encourage ownership of learning the
    seven principles discussed above.
  • Case examples from REAP applying these
    conditions/ principles

16
REAP Example 1 Mechanical Engineering(personal
response systems)
17
Problems identified
  • Conceptual misunderstandings even after
    graduation (e.g. concept of force)
  • Passive learning in classroom due to larger
    numbers
  • Evidence of low levels of student motivation
    attendance
  • Difficult to develop a sense of community amongst
    learners
  • Retention issues (20)

18
Course Redesign
  • Looked worldwide for the best solution
  • Focus on teaching core concepts
  • Carefully constructed student workgroups
  • Introduced personal response systems in lecture
    sessions (to facilitate peer discussion) as
    promoted by Mazur at Harvard
  • Later linked this to online testing
  • And to online homework system

19
Personal Response System (PRS)
PRS was developed by Professor Nelson Cue at Hong
Kong University of Technology and has now been
adopted by hundreds of educators worldwide
20
Benefits
  • PRS promotes reflection on and peer discussion of
    reasoning behind difficult concepts
  • Self, peer and tutor feedback in a single
    classroom session
  • Improved understanding in standardised
    engineering tests (e.g. force concept inventory)
  • Culture of collaborative learning established
  • Students report enhanced satisfaction (fun!)
    compared to traditional lecture classes.
  • High levels of time on task in class
  • Dropout reduced from 20 to 3

21
Some results
22
The student experience
  • Some student comments
  • in this class everybodys involved, you have to
    think about whats being said
  • you are learning from people around you its a
    language you can understand
  • you feel you are keeping pace with the class
    and that everyone is learning together

23
Relation to Gibbs Simpsons four assessment
conditions
  • Web-based assessment tasks (MCQs and problem
    solving exercises) keep students engaged in
    out-of-class activities and EVS encourages
    engagement in class (condition 1)
  • Activities are distributed across topics and
    weeks (condition 2)
  • EVS tasks are designed to deepen learning as
    conceptual understanding increases (condition 3)
  • EVS activities clearly communicate requirements
    and there is a progressive increase in challenge
    (communicates clear and high expectations,
    condition 4)

24
Relationship to seven principles
  • Learning goals clarified through iterative cycles
    of tutor presentation, tests and retests using
    MCQs in class (Principle 1)
  • Reflection/self-assessment triggered through
    bar-chart presentation and by online tests
    (Principle 2)
  • Teachers provide feedback at end of EVS
    concept-test sequence (Principle 3)
  • Both teacher-student and peer dialogue occur in
    EVS interactive sessions (Principle 4)
  • The focus on learning goals rather than
    performance goals in class and the staged
    difficulty of concept tests encourage
    motivational (Principle 5)
  • The continuous cycle of tests, retests and
    feedback ensures that students can use the
    feedback immediately (Principle 6)
  • Online MCQ tests and student performance in EVS
    lectures provide a range of feedback information
    that tutors can use adjust teaching to student
    needs (Principle 7)

25
Other uses of PRS
  • Diagnostic testing at beginning of year
  • Community building induction activities and
    sharing data about student cohort
  • Revision of lecture materials in class
  • Formal examinations using multiple choice
    questions
  • Evaluation data reactions to course
  • Experiments where human responses being tested
    (voting in politics, visual illusions,
    questionnaires)
  • Been used across a range of disciplines worldwide
  • See resources section of www.reap.ac.uk website
    and D. Banks Audience Response Systems in Higher
    Education Applications and Cases, Information 
    Science Publishing, Hershey 2006

26
Psychology
  • 560 first year students
  • Mixture of psychology majors (130) and those
    taking psychology only for one year (430)
  • 6 topic areas, 48 lectures, 4 tutorials, 12
    practicals
  • Assessment 2 x MCQs (25), tutorial attendance
    (4), taking part in experiment (5), essay exam
    (66)

27
Problems identified
  • No practice in writing skills but required in the
    exam
  • More detail provided in lectures than mentioned
    in exams (not enough independent reading)
  • No feedback except on MCQs (percent correct)
  • Didnt want to increase staff workload
  • Wanted to improve overall exam marks
  • And standard of entrant to second year

28
Discussion point
  • What would you do to improve the student
    experience in first year psychology?
  • You can use any technology (or combination of
    technologies) but you must consider costs and
    staff time constraints

29
Psychology Redesign
  • Discussion board in WebCT
  • Students in 85 discussion groups of 7-8, same
    groups throughout year
  • Series of online tasks
  • Lecture on Monday, Friday lecture abolished
  • Students to discover for themselves in
    collaboration with their group what would have
    been the topic of the Friday lecture

30
Structure of group tasks
  • 6 cycles of 3 weeks (one cycle x major course
    topic)
  • First week light written task (e.g. define
    terms) 7 short answers (all answer)
  • Second week guided reading
  • Week three heavy written task students answer
    guided questions and then collaborate in writing
    a 700-800 word essay.
  • Within each week
  • The Monday lecture introducing material
  • Immediately after lecture, task posted online
    for delivery the following Monday
  • Model answers (selected from students) posted for
    previous weeks task

31
The teaching role
  • Participation in the discussions was compulsory
    but not marked (this year there is 2 mark for
    participation)
  • The course leader provided general feedback to
    the whole class often motivational
  • He encouraged students to give each other
    feedback
  • And he selected the model answers
  • The group discussions were not moderated
  • Around 8 teaching assistants monitored the
    discussions and reported non-participation to the
    teacher

32
  Online Project 1 Classical Conditioning
Phenomena.  
 Each Group Member should read the Passer chapter
from the beginning to at least as far the section
which begins Applications of Classical
Conditioning. Satisfy yourself that you can
answer EACH of the questions below. Then agree as
a group who will answer what.  Project 1 is to
answer these questions as fully as you can   1)
What type of response is susceptible to Classical
Conditioning?   2) Why does Extinction
occur?   3) What is Spontaneous Recovery?   4)
What does the phenomenon of Spontaneous Recovery
tell us about the nature of Extinction in
Classical Conditioning?   5) What is
Generalisation?   6) What is Discrimination?   7)
What is Higher Order Conditioning?  
33
Project 9 An example of heavy task
  • The Task 800 word essay
  • Assess the strengths and weaknesses of Freuds
    and Eysencks theories of personality. Are the
    theories incompatible?
  • readings suggested
  • questions provided all should try
  • and advice on how to divide task given

34
Benefits
  • Written responses of an exceedingly high standard
    (sometimes surpassing 3rd year)
  • Spontaneous online discussions about learning and
    leaner responsibility
  • High levels of motivation, atmosphere in class
    improved
  • Some students burdened by workload easily
    detected
  • Some requested to move groups (5 groups)
  • Online interactions showed powerful scaffolding
  • Interaction and feedback possible with 560
    students
  • Easy for tutors to monitor participation
  • Peer feedback and self feedback (model answers)
    harnessed
  • Improved mean exam performance (up from 51-57)

35
Has it worked?
36
(No Transcript)
37
(No Transcript)
38
(No Transcript)
39
Online postings/interaction
  • 24,362 messages posted by groups
  • Average number of postings per student 44.3
  • Most messages posted by single student 324
  • 1067 postings to general open discussion forum
  • Accumulated time students spent in psychology
    site was 1 year 37days 11 hours 56 mins
  • Students set up online study groups for other
    subjects
  • Structured tasks online triggered important
    social-cognitive processes

40
Relation to the Gibbs Simpsons four
assessment conditions
  • Tasks require significant study out of class
    (capture sufficient study time, condition 1)
  • They are distributed across topics and weeks (are
    spread out evenly, condition 2)
  • They move students progressively to deeper levels
    of understanding (productive/ deep learning,
    condition 3)
  • There are explicit goals and progressive increase
    in challenge (communicates clear and high
    expectations, condition 4)

41
Relation to 7 feedback principles
  • Standard format and model answers provide
    progressive clarification of expectations (clear
    goals, principle 1)
  • Students encouraged to self-assess against model
    answer (self-assessment, principle 2)
  • Course leader provides motivational and
    meta-level feedback and selects model answers
    (teacher feedback, principle 3)
  • Online peer discussion aimed at reaching
    consensus about response (dialogue, principle 4)
  • Staged complexity, focus on learning not just
    marks, use of students model answers (motivation,
    principle 5)
  • Repeated cycle of topics and tasks (closing gap,
    principle 6)
  • VLE captures all interactions allowing course
    leader to monitor progress and adapt teaching
    (shaping teaching, principle 7)

42
What can we learn from these case studies?
  • Use of simple technologies (PRS, quiz/survey
    tool, discussion board)
  • Considerable thought gone into the learning
    design which is transferable
  • The drivers were learning improvements rather
    than technology (context of use)
  • Key finding across studies was need to balance
    structure and learner control
  • Also important is the way that the social and the
    academic processes are shown to be mutually
    supportive

43
Discussion point
  • Consider your own classes/modules/programmes
  • Identify one or two ideas that you might use from
    todays presentation in the redesign of your
    teaching.
  • Identify any questions raised in your discussion
    groups
  • Plenary report back
  • Be prepared to share an idea with others in the
    plenary and/or
  • A question worth asking that might illuminate
    some important issue

44
QAA Scotland Project Assessment and the First
Year Experience
  • Use assessment and feedback as lever for
    transformation of the student experience
  • Develop in students the ability to monitor,
    evaluate and regulate their own learning
    (engagement-empowerment)
  • Foster learning groups and communities in the
    first year and beyond (academic-social
    integration)

45
The Concepts
  • Balancing engagement (time effort on task) and
    empowerment (opportunities to take control and
    manage own learning)
  • Academic and social integration into the
    university use the academic to support the
    social and vice versa

46
EMPOWERMENT/ SELF-REGULATION
SOCIAL EXPERIENCE
ACADEMIC EXPERIENCE
ENGAGEMENT
Figure 1 Assessment and feedback practices
dimensions of implementation
47
  • Good formative assessment and feedback practices
    should
  • Help clarify what good performance is (goals,
    criteria, standards)
  • Encourage time an effort on challenging
    learning tasks
  • Deliver high quality feedback information that
    helps learners self-correct
  • Provide opportunities to act on feedback
  • Ensure that summative assessment supports
    formative learning processes
  • Encourage interaction and dialogue around
    learning (peer, teacher-student)
  • Facilitate the development of self-assessment and
    reflection in learning
  • Give choice in the topic, method, criteria,
    weighting or timing of assessments.
  • Involve students in decision-making about
    assessment policy and practice
  • Support the development of learning groups and
    communities
  • Encourage positive motivational beliefs and
    self-esteem
  • Provide information to teachers that can be used
    to help shape their teaching

48
Guidelines for Implementation (1)
  • Implement the principles
  • A single principle or many
  • Active involvement of students
  • Tight-loose maintain fidelity to the principles
    (tight) but encourage disciplines develop their
    own techniques of implementation (loose)

49
EMPOWERMENT/ SELF-REGULATION
  • Students create criteria
  • Students add own criteria
  • Students identify criteria from samples of work
  • Exemplars of different performance levels
    provided
  • Students rephrase criteria in own words
  • Provide document with criteria

ACADEMIC EXPERIENCE
SOCIAL EXPERIENCE
ENGAGEMENT
Figure 2 Assessment and feedback practices
dimensions of implementation
50
EMPOWERMENT/ SELF-REGULATION
SOCIAL EXPERIENCE
ACADEMIC EXPERIENCE
6.Encourage interaction and dialogue around
learning (peer and teacher-student)
2. Encourage time effort on challenging
learning tasks

ENGAGEMENT
51
EMPOWERMENT/ SELF-REGULATION
Students create MCQs and feedback for wrong and
right answers
6.Encourage interaction and dialogue around
learning (peer and teacher-student)
Students self-assess using MCQ and provide
confidence ratings
ACADEMIC EXPERIENCE

SOCIAL EXPERIENCE
Students self-assess own performance using MCQs
ENGAGEMENT
52
Guidelines for Implementation (2)
  • Determine where ICT can add value
  • Clarify students responsibilities
  • Align responses to National Student Survey to the
    assessment principles
  • Alternate solo and group work
  • Evaluate change (PIs)
  • Consider programme coherence
  • Use principles as quality enhancement tool
  • Share your learning and designs
  • Quality enhancement

53
Transforming AssessmentA blueprint for change
in the first year
  • TABLE OF CONTENTS
  • Recommendations how to implement at module,
    course and institutional level
  • A literature review how assessment and feedback
    can support empowerment and integration
  • 12 principles of assessment and feedback practice
    ( questions to enhance quality)
  • 100 examples and case studies illustrating
    implementation across the disciplines
  • (see also www.reap.ac.uk)

54
Some publications
  • Nicol, D (2008), Transforming assessment and
    feedback Enhancing integration and empowerment
    in the first year, to be published by Quality
    Assurance Agency, Scotland (June)
  • Nicol, D (in press), Assessment for learner
    self-regulation Enhancing achievement in the
    first year using learning technologies,
    Assessment and Evaluation in Higher Education,
  • Nicol, D (2007), Laying the foundation for
    lifelong learning cases studies of technology
    supported assessment processes in large first
    year classes, British Journal of Educational
    Technology, 38(4), 668-678
  • Nicol, D (2007) E-assessment by design using
    multiple-choice tests to good effect, Journal of
    Further and Higher Education.31(1), 53-64.
  • Nicol, D. Milligan, C. (2006), Rethinking
    technology-supported assessment in relation to
    the seven principles of good feedback practice.
    In C. Bryan and K. Clegg, Innovations in
    Assessment, Routledge.
  • Nicol, D, J. Macfarlane-Dick (2006), Formative
    assessment and self-regulated learning A model
    and seven principles of good feedback practice,
    Studies in Higher Education, 31(2), 199-218.
  • Boyle, J.T. and Nicol, D.J. (2003) Using
    classroom communication systems to support
    interaction and discussion in large class
    settings, Association for Learning Technology
    Journal, 11(3), 43-57
  • Nicol, D.J. and Boyle, J.T. (2003), Peer
    interaction and class-wide discussion a
    comparison of two interaction methods in the
    wired classroom, Studies in Higher Education,
    28(4), 457-73
  • See also www.reap.ac.uk
About PowerShow.com