Computerised Free-text Assessment of Key Feature Problems - PowerPoint PPT Presentation

About This Presentation
Title:

Computerised Free-text Assessment of Key Feature Problems

Description:

Computerised Free-text Assessment of Key Feature Problems Adrian Molyneux Steve Capey School of Medicine Project aims Fulfil student requests for more formative ... – PowerPoint PPT presentation

Number of Views:106
Avg rating:3.0/5.0
Slides: 16
Provided by: AdrianM1
Category:

less

Transcript and Presenter's Notes

Title: Computerised Free-text Assessment of Key Feature Problems


1
Computerised Free-text Assessment of Key Feature
Problems
  • Adrian Molyneux
  • Steve Capey
  • School of Medicine

2
Project aims
  • Fulfil student requests for more formative
    assessment
  • Fulfil need for practice of summative exam
    formats
  • Provide quality feedback
  • Achieve this without a heavy marking load placed
    on staff

3
Key-feature problems (KFPs)
  • an innovative assessment format that tests
    clinical decision making skills by focussing only
    on the critical steps - or 'key features' - of
    each decision, thereby increasing the number of
    problems that can be answered in an examination.
  • - Liz Farmer, Royal Australian College of GPs

4
SCENARIO - Jennifer Evans, a 38-year-old
statistician, presents to the accident and
emergency department with a two hour history of
sudden onset of leftsided pleuritic chest pain
shortness of breath and an episode of
haemoptysis. She had returned from a conference
in Malaga on a flight 48 hours earlier. On
examination she was found to be slightly febrile,
tachycardic and in atrial fibrillation. Her
blood pressure is 98/64mmHg. Examination of the
chest showed decreased expansion on the left and
localised decreasing breath sounds with a pleural
rub. A plain radiograph of the chest shows a
very small trace of fluid in the left
costophrenic angle to the lungs are clear. A
presumptive diagnosis of pulmonary embolism is
made.
5
Example questions from scenario
  • 1. What initial investigations should be
    requested? Name only two (2)
  • 2. Long distance travel is considered a minor
    risk factor for venous thromboembolism what are
    the major risk factors? Name only four (4)
  • 3. What is the mechanism of action of heparin?
  • 4. Shortly after the heparin was commenced,
    Jennifer was started on warfarin. What
    endogenous compound does warfarin inhibit to
    produce its anticoagulant effect? List only one
    (1)
  • 5. How is warfarin therapy monitored?

6
Research questions
  • How do we train a computer system to mark an
    open-ended freetext response?
  • How to deliver the assessment
  • How to feed back the results to students
  • How to author succinct KFPs
  • Can we develop a method for the most efficient
    creation of a computer-marked KFP?

7
Methods
  • Generate 10 KFPs and marking guidelines
  • Generate computer markscheme
  • Build web interface
  • Recruit students and run exam
  • Computer-mark responses
  • Tutor-mark responses
  • Refine markschemes (iterative process)
  • Statistical analysis

8
(No Transcript)
9
Exam delivery system and results
  • Exam delivery
  • Results

10
Computer Tutor comparison
11
Markscheme improvement
  • Make first attempt at markscheme and run exam
  • Examine student answers for further valid
    responses
  • Include these in computer markscheme and mark a
    second time
  • Examine computer markscheme for badly-performing
    questions and adjust (maybe loosen). Mark a
    third time.

12
Run Number Mean Mark of Human Mark
1 0.9 39
2 1.1 47
3 2.0 86
13
Conclusions practical limitations
  • Succinct answers should be encouraged perhaps
    limit text entry
  • Expect unexpected answers from students and
    markscheme modification
  • Standardisation of human marking
  • Adherence of humans to marking guidelines
  • Evolution of computer markschemes to cover the
    whole range of student answers

14
Conclusions - positives
  • Computer marking ideal for questions with narrow
    range of possible responses
  • Reduce marking hours spent on formative
    questions, whilst providing quality feedback in a
    realistic examination format
  • Perfect for low-stakes, on-demand assessment

15
Future direction of work
  • Run exams with full year groups
  • Refine question authoring
  • Evaluate other freetext question formats
    currently used within the medical school
  • Investigate the possibility of distributing tests
    asynchronously throughout the curriculum
Write a Comment
User Comments (0)
About PowerShow.com