Title: TENCompetence Assessment Model, Related Tools and their Evaluation Milen Petrov, Adelina AleksievaPe
1TENCompetence Assessment Model, Related Tools and
their Evaluation Milen Petrov, Adelina
Aleksieva-Petrova, Krassen Stefanov, Judith
Schoonenboom, Yongwu Miao
2Content
- TENCompetence Assessment Model
- Assessment Editor
- Assessment Run-time system
- Evaluation plan
- Main Results
- Conclusions
3Main goals
- Design of an assessment methodology for
competences, through analysis of the modern
assessment methods, selection of proper methods
and tools and design of basic assessment
activities. - Outline the major principles for planning and
design of effective assessment and provides a
framework and guidelines for the design of the
unit of assessment.
4TenCompetence Assessment Model
- Describes the life-cycle of the assessment
- Specified as formal specification using UML
diagrams - Optimised for competence development assessments
- Developed through simplification of the existing
OUNL/CITO Model
5Assessment model components
- Assessment design
- Item construction
- Assessment construction
- Assessment run
- Response rating
6(No Transcript)
7The reasons for simplifying OUNL/CITO assessment
model
- The OUNL/CITO model is extensive and complex,
aiming for completeness in its coverage of all
forms of assessment. - The TENCompetence Assessment Model, as part of
the TENCompetence Domain Model, needs not to
duplicate its components
8Proof-of-concept tools
- Needed to validate the model
- Two competence assessment methods demonstrated
360 degree feedback and Portfolio assessment - Assessment Authoring tool
- Assessment run-time tool
- Developed as Java Eclipse plug-ins
9Relation between TENCompetence assessment model,
QTILD specifications and first
proof-of-concept-tools
10TENCompetence Assessment Data Model
- Why needed to provide the needed semantics for
the Assessment specification - Specified as a XML schema
- Used to store the information about any specific
details regarding any stage of the assessment
process - Used as an output from editing and input for the
run-time
11Functionality of the tools
12Assessment Authoring tool
- Covers the following three phases of the
TENCompetence Assessment Model Assessment
design, Item construction and Assessment
construction - Implemented as 360 degree feedback competence
assessment method editor - Results in an XML Schema according to the
TENCompetence Assessment Data Model
13(No Transcript)
14The Authoring tool components
- Design phase Editor - define the blueprint of the
assessment - Question editor (item construction phase) -
create and edit different type of questions
(demonstration items, construction items, and
selection items) - Assessment Test Editor - combine assessment
architect and assessment items
15Assessment run-time tool
- Runtime environment for playing any
non-traditional method of assessment - Supporting the TENCompetence Assessment data
model - Implemented to support the Portfolio Assessment
method - Assessment run is based on an XML Schema
according to the TENCompetence Assessment Data
Model
16(No Transcript)
17The Runtime tool components
- Assessment run-time simulator loads and perform
non-traditional forms of assessment (like
portfolio assessment peer review, etc) - Response processing for tracking the results
from the assessment
18Evaluation goals
- Is the model able to be used for the
implementation of different competence assessment
methods - Are the tools capable to demonstrate the
applicability of the model - What is the complexity of these tasks
19Evaluation Methodology
- Evaluate the functional quality of the tools
(test case review checklist) - Rate the interface and usability of the systems
(end user questionnaire) - Evaluate the software code quality and complexity
(expert evaluation checklist)
20Users involved
- Students for the usability testing
- Educational technology experts for testing the
functional quality of the tools - Software technology experts for the evaluation of
the code quality and complexity
21Design principles of ISO 9241 (Part 10)
- suitability to task
- self explanatory
- controllability
- conformity with user expectations
- error tolerance
- suitability for individualisation
22Part of the given user questionnaire
23Evaluation procedure
- Step 1 Download the evaluation bundle (user
guide and assessment instruments) - Step 2 Download the corresponding
proof-of-concept tool - Step 3 Un-package and install the software
- Step 4 Work with the software
- Step 5 Fill-in the assessment instrument
- Step 6 Return the filled assessment instrument
24Details of the evaluation process
25ISO Evaluation Results
26Main Evaluation Results
- Expert software code evaluation results good
quality of the code but very high complexity if
needed to go for real life tools - Functional quality results very good functional
quality - Usability evaluation results overall good
quality, but several errors and issues identified
27Evaluation Conclusions
- The tools proved the applicability of the model
- The Model has sufficient modelling power
- The complexity needed to go from proof-of-concept
to real-life tools is very high and risky
28Next steps
- 1) Further research needed into improving the
mapping algorithms which are used to carry out
the transformation to the data model - 2) Revise and extend the TENCompetence Assessment
Data Model, using elements with the functionally
similar to those used in IMS LD and QTI
29Questions?