Learning From Assessment: Evaluating the Benefits of DALI (Diagnostic Assessment Learning Interface)

1
Learning From Assessment Evaluating the Benefits
of DALI (Diagnostic Assessment Learning Interface)

• Hershbinder Mann Guinevere Glasfurd-Brown,
• University of Essex
• 2006

2
Background and Context

• The DALI Project is a Teaching and Learning
  Innovation Fund (TALIF) pilot study at the
  University of Essex.
• Development of the student portal and online
  formative assessment resources as part of the
  SPRInTA Project.
• Online formative testing in three large
  Departments, primarily to address specific
  objectives
• large class teaching poor performance, high
  drop-out rate, lack of engagement with case study
  materials
• threshold testing
• improving feedback (and opportunities for
  feedback) to students (Nat Student Survey SSS)

3
Issues that emerged from past practice

• Student experience
• Aims of the learner? (motivation hoe might we
  improve engagement?)
• Is there a gap between what students expect from
  online formative assessment and what they
  get/experience?
• How can we encourage a dialogue between staff and
  students on assessment?
• How might this inform the guidance we provide to
  staff?

4
Background and Context
Learners have different priorities, preferences
and approaches to learning, and different
requirements for support. The learning
environment needs to reflect these differences.
Understanding how different learners experience
the tasks, resources and services offered to them
is an important precursor to developing
effectively personalised systems. -
Understanding My Learning, JISC
5
DALI Project

• DALI tool (issues) enables students to revisit
  and query assessment data and identify areas of
  weakness (related to specific learning outcomes
  at topic level)
• Survey of students
• Assessment Think Tank student focus group
• iLAB session which brought staff and students
  together to discuss assessment.

6
DALI

• Diagnostic Assessment Learning Interface
• DALI is an add-on to Perception. It focuses on
  the provision of assessment feedback and provides
  students with information aimed at giving them
  greater control over their own learning.
• The interface
• allows a student to select the score level they
  wish to achieve
• highlights the last topic-based feedback
  provided to the learner
• provides statistics associated with any selected
  topic and
• displays topic descriptions as learning
  objectives.

7
Using DALI
8
Aims

• To identify the factors that affect the uptake
  of computer-aided assessment among students
• Strengths and weaknesses from a student
  perspective
• The importance of feedback
• Good / bad feedback
• What should feedback do?
• How can this be achieved?
• The role of DALI
• Guidelines for users

9
Data Collection

• Participation from three University Departments
• Accountancy and Financial Management, Electronic
  Systems Engineering, and Biological Sciences.
• Online student survey
• Student focus group
• Staff and student iLab meeting

10
DALI Project Survey - Results

• Identifying my strengths and weaknesses in
  assessment is important to me.

• It is important for me to be able to track my own
  progress.

• The online practice tests are a useful learning
  tool.

11
DALI Project Survey - Results

• The tests help me learn more effectively.

• I would take more tests if available.

• I pay little attention to answering the questions
  correctly.

12
DALI Project Survey - Results

• I use the tests mainly for the feedback.

• The feedback is what I expected.

• The feedback on the questions could be improved.

13
DALI Project Survey - Results

• The survey also invited respondents to add
  additional comments relating to feedback and to
  online assessment more generally.
• These comments were categorised and placed in one
  of the following groups
• Feedback, Knowledge/Learning, Revision/Planning,
  Accessibility, Technology, Convenience, Question
  Format, Self-Test, Time, Motivation, Memory,
  Relevance, Other.

14
DALI Project Survey - Results
15
DALI Project Survey Results
16
Think Tank

• As a supplement to the survey, a focus group was
  assembled. Participants were asked about their
  experiences of online assessment.
• Questions concentrated on likes/dislikes, reasons
  for doing the tests, and views on the feedback
  received.

17
Think Tank Results
18
Focus Group Findings

• The focus group corroborated the survey findings.
• Positives
• Students perceive the benefits of using online
  assessment in relation to learning,
  comprehension, and revision, and were pleased to
  get instant feedback.
• Negatives
• Feedback is vague, access is difficult, questions
  are not engaging enough, and the questions are
  not relevant.

19
iLab Session Key Results

• Student comments
• Revision
• Personal strengths and weaknesses
• QMP is not used to its full potential
• Feedback could be enhanced to make it more
  relevant to the individual
• Staff comments
• Time, workload, and support
• Developing assessments and question banks is
  time-consuming as is writing effective feedback
• The ability to extend and update tests is very
  useful

20
Conclusions

• Project findings have been used to inform a
  good practice staff handbook on online
  formative assessment.
• The project encountered difficulties with the
  DALI interface and this is under investigation.
• The project was instructive in highlighting the
  importance students place upon effective feedback
  and the potential to personalise their learning.
• The University is developing an intelligent
  assessment environment which has the ability to
  generate personalised learning profiles.