Phil Davies

1
The Automatic Generation of Marks for Marking
within the Computerised Peer-Assessment of Essays

• Phil Davies
• School of Computing
• University of Glamorgan Super U

2
Computerised Peer-Assessment

• CAP System
• Permits students to mark comment the work of
  other students .
• Also self-assess used as a standard of
  expectation
• Internet, not Web-based system (developed in
  Visual Basic / Access)

3
(No Transcript)
4
Having done the marking, what next?

• Students should receive feedback
• What feedback?
• Marks
• Comments
• Which is most important?

5
(No Transcript)
6
AUTOMATICALLY EMAIL THE MARKER .. ANONYMOUS
7
What should the marker do?Reflect

• Look at essay again
• Take into account the essay owners comments
• Further clarification (if it is needed, then is
  this a black mark against the marker?)
• Try to appease the essay owner?
• Modify mark based upon reflection?
• Give more feedback

8
(No Transcript)
9
Must be rewarded for doing the mark for marking
process

• How to judge?
• Standard of expectation (self-assessment)
• Marking consistency
• Commenting, quality, measure against mark
• Discussion Element
• Need for additional comments black mark?
• Reaction to requests / further clarification

10
Does this student deserve a good mark?
ESSAY Compensated Median Mark?? Student Mark Difference
W 65 67 2
X 48 54 6
Y 53 57 4
Z 61 59 -2
2 on average
11
Standard of ExpectationSelf-Assess
68Peer-Assessed as 58
ESSAY Compensated Median Mark Student Mark Difference
W 65 67 2 (8)
X 48 54 6 (4)
Y 53 57 4 (6)
Z 61 59 -2 (12)
X 2 (7.5)
12
(No Transcript)
13
How easy is it to do?

• Statistically fairly easy to create a mark for
  marking based upon marks
• Take into account high and low markers
• Standard of expectation
• Consistency judge against final mark awarded
  for an essay (compensated median)
• What about the comments?

14
Feedback Index

• Produce an index that reflects the quality of
  commenting
• Produce an average feedback index for an essay
• Compare against marker in a similar manner to
  marks analysis
• Where does this feedback index come from and is
  it valid?

15
The way to get the feedback index?

• Develop an application??
• C-Rater?
• Spelling Mistakes
• Similar Meanings?
• That was cool
• Really Choc
• Really Good Essay
• Manually

16
(No Transcript)
17
(No Transcript)
18
(No Transcript)
19
(No Transcript)
20
(No Transcript)
21
(No Transcript)
22
(No Transcript)
23
(No Transcript)
24
Differences from actual F/Index Mark
Score
DAI SMITH ESSAY 1 4.5 3.8
3/10
ESSAY2 -3.0
8.5 5/10
. .
FINAL MARK AWARDED 5/10
25
Time Consuming?

• Can we formulate the marking process
• Take away need for quantification process of
  analyzing comments
• Is it still peer-assessment if the students are
  told what to say?

26
(No Transcript)
27
STUDENT FRED REFERENCES Positive Negative
.
Personal Valuation 5, 3, 2, 1 3, 1, 2
28
Some points outstanding

• What should students do if they identify
  plagiarism?
• Is it ethical to get students to mark the work of
  their peers?
• Is a computerised solution valid for all?
• At what age / level can we trust the use of peer
  assessment?
• How do we assess the time required to perform the
  marking task?

29
The Automatic Generation of Marks for Marking
within the Computerised Peer-Assessment of Essays

• Student Acceptance
• Anonymity / Discussion / High Low Markers
• Tangible Reward
• Im more concerned with rewarding higher order
  skills
• Getting there