The Quality of Peer-Feedback in the Computerised Peer-Assessment of Essays? The case for awarding

1
The Quality of Peer-Feedback in the Computerised
Peer-Assessment of Essays? The case for
awarding marks for marking based upon peer
feedback not marks.
OR Lecturer getting out of doing marking

• Phil Davies
• School of Computing

2
Need for assessment?

• As tutors we are trying to separate the sheep
  from the goats via the assessment process.
• This can often be difficult with the time
  constraints imposed on tutors, in what Chanock
  describes as more goat-friendly times (Chanock,
  2000).
• Feedback against time!

3
Defining Peer-Assessment

• In describing the teacher ..
• A tall b, so he was. A tall thin, mean
  b, with a baldy head like a lightbulb. Hed
  make us mark each others work, then for every
  wrong mark we got, wed get a thump. That way
  he paused we were implicated in each others
  pain

McCarthys Bar (Pete McCarthy, 2000,page 68)
4
Why Peer-Assessment

• Working together to learn (and be assessed)
• Perceptions
• Staff gt Lecturer getting easy life not marking
• Studentgt Lecturer getting easy life not marking
• Student Awareness of Benefits . IMPORTANT
  TANGIBILITY
• Success dependant upon scalability
  (computerisation) .. Student Numbers have risen
  dramatically since 1991 without a concomitant
  increase in resources (Pond et al, 1995)

5
Computerised Peer-Assessment

• CAP System
• Permits students to mark comment the work of
  other students . (normally 6-8)
• Also initial self-assess stage (reflection)
  used as a standard of expectation
• Internet, not Web-based system (developed in
  Visual Basic / Access)

6
(No Transcript)
7
Having done the marking, what next?

• Students should receive feedback
• What feedback?
• Marks
• Comments
• Which is most important?
• To students or staff

8
(No Transcript)
9
AUTOMATICALLY EMAIL THE MARKER .. ANONYMOUS
10
What should the marker do?Reflect

• Look at essay again
• Take into account the essay owners comments
• Further clarification (if it is needed, then is
  this a black mark against the marker?)
• Try to appease the essay owner?
• Modify mark based upon reflection?
• Give more feedback

11
(No Transcript)
12
Exercise One

• In the top box what are the inherent advantages
  and disadvantages of peer-assessment
• Middle Box Mark (/10) and comment to support
• Bottom Box Mark the marker /10 and comment to
  support
• Now give the work back!

13
Must be rewarded for doing the mark for marking
process .. Based on quality

• How to judge?
• Standard of expectation (self-assessment)
• Marking consistency
• Commenting, quality, measure against mark
• Discussion Element
• Need for additional comments black mark?
• Reaction to requests / further clarification

14
(No Transcript)
15
How easy to get an automated mark for marking?

• Statistically fairly easy to create a mark for
  marking based upon marks
• Take into account high and low markers
• Standard of expectation
• Consistency judge against final mark awarded
  for an essay (compensated median)
• What about the comments?

16
Feedback Index

• Produce an index that reflects the quality of
  commenting
• Produce an average feedback index for an essay
  (also compensated?)
• Compare against marker in a similar manner to
  marks analysis
• Where does this feedback index come from and is
  it valid?

17
The way to get the feedback index?

• Develop an application??
• C-Rater?
• Spelling Mistakes
• Similar Meanings?
• That was cool
• Really Choc
• Really Good Essay
• Manually

18
(No Transcript)
19
Commonality!!

• In the 67 essays that were marked
• Only 96 comments
• 44 positive and 56 negative
• Highly critical if something not explained
  properly (21 of total comments (of which 73
  were negative)
• Comments grouped into 10 categories
• Need to QUANTIFY these comments .. Feedback index
• Create a database holding positive negative (by
  category)

20
(No Transcript)
21
(No Transcript)
22
(No Transcript)
23
(No Transcript)
24
(No Transcript)
25
(No Transcript)
26
-7 -6 -5 -4 -3 -2 -1 -0 0 1 2 3 4
34 34 51 56 52 56 61 63 70 64 68 70
39 52 61 46 57 58 62 72 65 70
44 45 52 54 66 59 62 69 68
34 35 57 66 65 65 66 69
56 59 68 64 63 71
61 59 66 66 66 66
55 48 60 69 68
61 57 58 65 73
59 61 60
63
62
68
34 37.8 45.8 59.6 55.6 58.8 62.9 62.8 67.5 68 68 70
27
General points of note

• Main criteria used for peer-marking
• Explanations, conclusions, references, examples
• Better students were critical (upper quartile)
• Better understanding permitting criticism?
• Confidence?
• Hostility?
• Weaker student in cloud cuckoo land
  consistently
• Consistency 30-34, 35-39, 55-59 65-69 70-74
• Note 70-74 OVER-Comment Mark

28
Time Consuming?

• Can we formulate the marking process
• Take away need for quantification process of
  analyzing comments
• Is it still peer-assessment if the students are
  told what to say?

29
(No Transcript)
30
Exercise Two

• I think youve missed out a big area of the
  research
• Youve included a big chunk that you havent
  cited
• There arent any examples given to help me
  understand
• Grammatically it is not what it should be like
• Your spelling is atroceious
• You havent explained anything to me
• Youve directly copied my notes as your answer to
  the question
• Most of what youve said is wrong

31
(No Transcript)
32
STUDENT FRED REFERENCES Positive Negative
.
Personal Valuation 5, 3, 2, 1 3, 1, 2
33
Is it my job to teach students how to write
essays, etc?

• Assessment MUST be directed
• Why bother writing essays, doing exam questions,
  etc. doesnt relate to needs or learning
  outcomes of subject
• Post HND N-tier Assess the essays of the
  final year (last year)
• Preparation/Research Judge knowledge against
  last years results .. Both marks comments
• Mistake!!

34
Can peer-assessment benefit other subject areas?

• Java Programming with Coursemarker
• Stuart Lewis idea
• Students create a solution to a programming
  assignment
• Submission(s)
• Peer-Evaluate other solutions
• Comments Marks for Marking (weightings)

35
CM
Modula-2 Java C
CourseMarker
Computer Assisted Teaching and Assessment

STUDENT
TEACHER
CourseMarker Core
Exercise Developm. System
Student Exercise Environment

• assignments
• exercises
• notes
• questions

File Storage System

• exercise setup
• submission

• edit
• compile
• link
• run

feedback and mark

• test methods
• solution template
• marking scheme

Marking System
Evaluation System

• final mark
• position in class
• course statistics

• course statistics
• flagging-up of problem cases

immediate support
comments / questions
FEATURES

• UNIX (Linux), Windows, Mac, based all
  platforms
• Assessment of text I/O assignments only
  no marking of graphical output
• remote student / teacher access distance
  learning, open all hours

Advantages / Disadvantages
TEACHER
STUDENTS

• re-usability
• automated marking
• - fair
• - frees time
• plagiarism check

• steep learning curve
• difficult setup
• (but its getting easier)

• immediate feedback
• fast support

• additional
• overheads

36
PeerMarker Screen
37
Student while marking

• Exposure to different solutions
• Development of critical evaluative skills
• Useful experience of reading code for future
  employment situations
• Plagiarism? Good solution / No understanding

38
Student while reviewing feedback from peers

• Range of subjective marking
• Confirmation of objective automated marking
• Anonymous discussion between marker and marked

39
Current position

• Test system working
• Changes following beta test in progress
• Plans to try sample study again (at a more
  convenient time, and with added rewards!)
• Employ 2nd Placement Student
• Graphical Interface

40
Some Points Outstanding or Outstanding Points

• What should students do if they identify
  plagiarism?
• Is it ethical to get students to mark the work of
  their peers?
• Is a computerised solution valid for all?
• At what age / level can we trust the use of peer
  assessment?
• How do we assess the time required to perform the
  marking task?
• What split of the marks between creation marking