From Individual Ratings to Consensus Ratings of L2 Essays

1
From Individual Ratings to Consensus Ratings of
L2 Essays

• Vivian Lindhardsen
• Copenhagen Business School
• Denmark
• Ph.D. Student
• Work in progress

2
From Individual Ratings to Consensus
RatingsVivian Lindhardsen, Copenhagen Business
School

• What are the decision-making strategies that
  experienced raters employ when going through
  individual ratings and consensus ratings to reach
  a final score?

3
From Individual Ratings to Consensus
RatingsVivian Lindhardsen, Copenhagen Business
School

• 20 highly experienced raters
• 15 L2 essays (exit exam of upper secondary
  school, business line)

4
From Individual Ratings to Consensus
RatingsVivian Lindhardsen, Copenhagen Business
School

• Introspection
• Observation
• Retrospection

5
From Individual Ratings to Consensus
RatingsVivian Lindhardsen, Copenhagen Business
School

• Official criteria
• Volume, Structure, Content and use of source
  materials, Language, Style, Other/General
• Self-monitoring or validation strategies

6
From Individual Ratings to Consensus
RatingsVivian Lindhardsen, Copenhagen Business
School
Individual Ratings
First impression Scan for length, title, overall
structure
Thorough interpretation and assessment Read,
classify errors, edit text, summarize ideas, etc.
Preliminary score
7
From Individual Ratings to Consensus
RatingsVivian Lindhardsen, Copenhagen Business
School
Consensus Ratings
Exchange preliminary scores
If different or not pass, then
Validation Exemplify directly from text, define
or revise own assessment strategies, consider
exam or task requirements, compare essays,
deliberate own teaching practices
Final score
8
From Individual Ratings to Consensus
RatingsVivian Lindhardsen, Copenhagen Business
School

• MG Essay 6. I gave it a 9
• GP Well, I actually gave it a 10 because the
  language is fluent and he is very good at using
  the source texts. And it is logical. Impressive
  vocabulary EMBARKING ON A CRUSADE.
• MG I see what you mean, but they have a couple
  of hours for this so I think there should be more
  than one page. We can expect that from them.
• GP Yes, it is difficult. I think that had it
  been my student I would have told him to write
  some more and keep on what hes doing. But
  youre right. 9 is probably more appropriate. It
  is too short.
• Note Average score for essay 6 8.9

9
Reference to Official Criteria
From Individual Ratings to Consensus
RatingsVivian Lindhardsen, Copenhagen Business
School
10
From Individual Ratings to Consensus
RatingsVivian Lindhardsen, Copenhagen Business
School

• Score preference
• 92 preferred consensus score
• 8 preferred individual score

11
From Individual Ratings to Consensus
RatingsVivian Lindhardsen, Copenhagen Business
School

• Dominance?
• To be finalized

12
From Individual Ratings to Consensus
RatingsVivian Lindhardsen, Copenhagen Business
School

• Implications
• Rater training
• Validation of consensus score