TENCompetence Assessment Model, Related Tools and their Evaluation Milen Petrov, Adelina AleksievaPe

1
TENCompetence Assessment Model, Related Tools and
their Evaluation Milen Petrov, Adelina
Aleksieva-Petrova, Krassen Stefanov, Judith
Schoonenboom, Yongwu Miao
2
Content

• TENCompetence Assessment Model
• Assessment Editor
• Assessment Run-time system
• Evaluation plan
• Main Results
• Conclusions

3
Main goals

• Design of an assessment methodology for
  competences, through analysis of the modern
  assessment methods, selection of proper methods
  and tools and design of basic assessment
  activities.
• Outline the major principles for planning and
  design of effective assessment and provides a
  framework and guidelines for the design of the
  unit of assessment.

4
TenCompetence Assessment Model

• Describes the life-cycle of the assessment
• Specified as formal specification using UML
  diagrams
• Optimised for competence development assessments
• Developed through simplification of the existing
  OUNL/CITO Model

5
Assessment model components

• Assessment design
• Item construction
• Assessment construction
• Assessment run
• Response rating

6
(No Transcript)
7
The reasons for simplifying OUNL/CITO assessment
model

• The OUNL/CITO model is extensive and complex,
  aiming for completeness in its coverage of all
  forms of assessment.
• The TENCompetence Assessment Model, as part of
  the TENCompetence Domain Model, needs not to
  duplicate its components

8
Proof-of-concept tools

• Needed to validate the model
• Two competence assessment methods demonstrated
  360 degree feedback and Portfolio assessment
• Assessment Authoring tool
• Assessment run-time tool
• Developed as Java Eclipse plug-ins

9
Relation between TENCompetence assessment model,
QTILD specifications and first
proof-of-concept-tools
10
TENCompetence Assessment Data Model

• Why needed to provide the needed semantics for
  the Assessment specification
• Specified as a XML schema
• Used to store the information about any specific
  details regarding any stage of the assessment
  process
• Used as an output from editing and input for the
  run-time

11
Functionality of the tools
12
Assessment Authoring tool

• Covers the following three phases of the
  TENCompetence Assessment Model Assessment
  design, Item construction and Assessment
  construction
• Implemented as 360 degree feedback competence
  assessment method editor
• Results in an XML Schema according to the
  TENCompetence Assessment Data Model

13
(No Transcript)
14
The Authoring tool components

• Design phase Editor - define the blueprint of the
  assessment
• Question editor (item construction phase) -
  create and edit different type of questions
  (demonstration items, construction items, and
  selection items)
• Assessment Test Editor - combine assessment
  architect and assessment items

15
Assessment run-time tool

• Runtime environment for playing any
  non-traditional method of assessment
• Supporting the TENCompetence Assessment data
  model
• Implemented to support the Portfolio Assessment
  method
• Assessment run is based on an XML Schema
  according to the TENCompetence Assessment Data
  Model

16
(No Transcript)
17
The Runtime tool components

• Assessment run-time simulator loads and perform
  non-traditional forms of assessment (like
  portfolio assessment peer review, etc)
• Response processing for tracking the results
  from the assessment

18
Evaluation goals

• Is the model able to be used for the
  implementation of different competence assessment
  methods
• Are the tools capable to demonstrate the
  applicability of the model
• What is the complexity of these tasks

19
Evaluation Methodology

• Evaluate the functional quality of the tools
  (test case review checklist)
• Rate the interface and usability of the systems
  (end user questionnaire)
• Evaluate the software code quality and complexity
  (expert evaluation checklist)

20
Users involved

• Students for the usability testing
• Educational technology experts for testing the
  functional quality of the tools
• Software technology experts for the evaluation of
  the code quality and complexity

21
Design principles of ISO 9241 (Part 10)

• suitability to task
• self explanatory
• controllability
• conformity with user expectations
• error tolerance
• suitability for individualisation

22
Part of the given user questionnaire
23
Evaluation procedure

• Step 1 Download the evaluation bundle (user
  guide and assessment instruments)
• Step 2 Download the corresponding
  proof-of-concept tool
• Step 3 Un-package and install the software
• Step 4 Work with the software
• Step 5 Fill-in the assessment instrument
• Step 6 Return the filled assessment instrument

24
Details of the evaluation process
25
ISO Evaluation Results
26
Main Evaluation Results

• Expert software code evaluation results good
  quality of the code but very high complexity if
  needed to go for real life tools
• Functional quality results very good functional
  quality
• Usability evaluation results overall good
  quality, but several errors and issues identified

27
Evaluation Conclusions

• The tools proved the applicability of the model
• The Model has sufficient modelling power
• The complexity needed to go from proof-of-concept
  to real-life tools is very high and risky

28
Next steps

• 1) Further research needed into improving the
  mapping algorithms which are used to carry out
  the transformation to the data model
• 2) Revise and extend the TENCompetence Assessment
  Data Model, using elements with the functionally
  similar to those used in IMS LD and QTI

29
Questions?