Teaching and Learning Applications related to the automated interpretation of ERDs

1
Teaching and Learning Applications related to the
automated interpretation of ERDs

• Pete Thomas, Kevin Waugh, Neil Smith
• Centre for Research in Computing
• Department of Computing
• The Open University, UK
• TLAD 2007, 2nd July, Glasgow

2
outline for this presentation

• background to our work on diagram understanding
• experiments marking ERDs
• applications using the ERD drawing tool and
  marker
• the marking tool (mark scheme setter)
• the student exerciser (revision tool)
• the markers assistant (tutor tool v0.1)
• where next.

3
diagram understanding
our approach

• A five stage process
• 1. segmentation
• 2. assimilation
• 3. identification
• 4. aggregation
• 5. interpretation

general diagram knowledge
convert raster-based images to general diagram
features lines, arrows, boxes, circles, arcs,
etc.

domain specific knowledge
identify minimal meaningful combinations of
features, combine into larger units and interpret
4
diagram understanding - marking
3
Feature-based diagram
Identified MMUs
4

5
Interpretation (grades)
(Aggregated MMUs) Meaningful Units
5
diagram understanding our approach minimal
meaningful units (MMUs)

• a minimal collection of diagram features that, in
  the given domain, have a recognisable
  interpretation.

Relationship
Entity Type 1
Entity Type 2
Super Type
SubType 1
Entity Type
Sub Type 2
6
diagram understanding our approach
aggregations of MMUs

• combinations of MMUs that together form
  meaningful units (MUs) with domain specific
  interpretations
• fixed aggregations

Entity Type 1
Entity Type 2
Entity Type 3

• rule-based aggregations

..
Entity Type 1
Entity Type 2
Entity Type 3
7
research context why use ERDs?

• the diagram notation is straightforward
• a range of styles of question
• closed with a single solution
• open with multiple acceptable alternative
  solutions
• a potentially wide range of student variations
  and errors (imprecision)
• large number of student answers available to us

however

• were confident that the approach will work with
  (most) graph-based diagrams.

8
the marking problem
Specimen Solution
Student Answer
How similar are they? Can we compute a
similarity measure comparable to that a human
marker would award? (for a given marking scheme)
9
what does the marking algorithm do?

• this is probably best described using a
  demonstration to show the first stages in the
  marking process
• (the marker support tool)

10
establishing confidence in the marking algorithm

• before we can start using the algorithm in
  teaching and learning support tools we need to be
  confident that it will work.
• we have a standard approach of using examination
  questions and student answers to evaluate the
  quality of the grading algorithm.
• the next few slides give the results of two major
  experiments we have undertaken so far.

11
experiment 1

• Question (3rd level database course, exam 2004)
• Consider the following scenario
• The following data requirements relate to the
  operation of a company, TrainingU, that provides
  training courses for the staff of client
  organizations.
• Each training course is assigned an identifying
  code and has a descriptive title. Each course is
  made up of one or more units. Each unit has a
  title and a unit code. The unit code is unique
  within the course containing that unit, but not
  unique across all units. Each unit is used in
  exactly one course.
• (a) Give an E-R model to represent these
  requirements. Your model should include an ER
  diagram, showing the degree and participation
  conditions for all relationships, entity types,
  any additional constraints not expressed

12
experiment 1
Sample solution Marks available 7

• Sample size 591 scripts development set 197
  diagrams
• test set 394 diagrams

13
experiment 2

• Question (3rd level database course, exam 2006)

Sample solution Marks available 7
14
experiment results

• Human Tool marking differences

Experiment 1 test set size 394
Experiment 2 test set size 72
15
experiment results

• frequency distribution of marks awarded

Experiment 1
Experiment 2
16
the marking (grading) tool

• grades student answers
• against one or more sample solutions
• using one or more marking schemes

other things we can do

• report on common student errors
• monitor human graders
• cross compare common unusual answers, this
  might have an application in plagiarism detection
• identify alternative plausible solutions and
  remark the scripts (dynamic adjustment of
  solution sets)

17
success

• the marking tool works well enough
• at this stage
  in its development
• our experience with the marking algorithm and
  drawing tool has lead us to develop several TL
  support tools
• work being funded by a HEFCE teaching fellowship
  held by Pete Thomas.

18
demonstrations

• students revision tool (mark student attempts at
  exercises, with question specific advice)
• markers support tool (redraws student answers, to
  support tutor-based marking)
• tutor tool (create exercises and marking schemes,
  for the above tools)
• marking tool (marks against marking schemes)

19
student revision tool
20
markers support tool
21
tutor tool
mark scheme encoding and revision tool question
setting
22
marking tool

• a very boring demonstration (sorry)
• the marking tool engine is at the heart of the
  student revision tool, tutor tool, etc.
• so
• we can see how improvements to the marking
  engine behave with large population marking tasks
  before implementing the changes in any additional
  tools.

23
where next.

• the majority of what has been done so far
  (experiments, pilot studies with student
  volunteers, etc.) has been about proving the
  technology the marking engine accuracy and tool
  usability.

need to consider a formal evaluation of the
revision tool, does the tool benefit the
students, how do the students use the tool,
etc. we will be including the revision tool as
a standard part of the 3rd level course in
relational databases during 2008 with a formal
framework for student feedback and an analysis
of student performance against tool use.
24
and thank you
Questions?