Diapositive 1

1
EARLI 11th Biennial conference 2005 electronic
Construction and Quality Control in Standardized
Testing the e-CQCST project Jean-Luc Gilles,
Sylvie-Anne Piette, Pascal Detroz, Salvatore
Tinnirello, Marc Pirson, Mbassa Dabo Hung
Lê http//www.smart.ulg.ac.be/earli-2005/e-CQCST.
ppt University of Liège - Belgium
2
I. Context

• Many European Countries have to face a growing
  number of higher education students
• The training professionals and the human
  resources managers of companies are often
  constrained to evaluate staff competencies, often
  acquired outside the schools
• There is a worldwide movement of the introduction
  of quality management in education and training
  activities

(Kaufman al., 1995 Segers Dochy, 1996
Ramsden, 1991 Nightingale ONeil, 1994 Zink
Schmidt, 1995 Castaigne, Gilles Hansen, 2001
Gilles, 2002 Gilles et Lovinfosse, 2004 Phelps,
2005)
3
I. Context

• The quality approach suggested by the SMART team
  in this context

4
II. Principals RD objectives of the e-CQCST
project

• To develop a web based platform which propose
  methods, procedures and tools for the management
  of standardized tests.
• To set up economic valorizations of the results
  of our research with supply of services like
• Audit of standardized testing systems
• Training and coaching of the users of the
  e-CQCST platform
• Docimological services for the improvement of
  standardized tests
• Test creation with the platform
• To propose a quality label for standardized
  testing

5
III. Method the CCQCST model
First step 1. ANALYSIS - list of the Points to
Evaluate (PE) - their Priorities (P) - and the
Categories of Performances (CP) of the learners
or trainees End product a test
specification table where Points to Evaluate
(PE) are crossed with Categories of Performances
(CP) ? binomials (PE x CP).
6
III. Method the CCQCST model
Second step 2. DESIGN The test developer
must design the Modalities of the Questions (MQ)
according to the Categories of Performances (CP)
previously defined at step 1. End products -
Binomials (CP x MQ) deriving from the Categories
of Performances (CP) associated with the
Modalities of the Questions (MQ) - Trinomials (PE
x CP x MQ) deriving from the binomials (PE x CP),
previously defined at step 1, assembled with
the binomials (CP x MQ) of this step 2 -
Selection of the trinomials which should be
linked to the questions of the test
7
III. Method the CCQCST model
Third step 3. QUESTIONS The test developer
selects or writes the questions. These questions
are linked to the trinomials objects selected for
the test at the former step. End product A
series of questions related to the content and
the performances to be test. These questions are
created or selected in a secured item bank
database.
8
III. Method the CCQCST model
3. Questions
Selection or writing of the questions which are
linked to the trinomials selected
Item bank
Fourth step 4. TRAINING The person in
charge of the test can choose to propose
trainings on the modalities of the questions
previously defined. End product blank tests
(paper-pencil or web based)
9
III. Method the CCQCST model
3. Questions
Selection or writing of the questions which are
linked to the trinomials selected
Item bank
Fifth step 5. TESTING The questions created
or selected at the third step are put in the
standardized test which is proposed to the people
who perform the test. End products - online
exams (in a secured environment)- questionnaires
linked with OMR
4. Training
on the modalities of the questions defined
Blank tests (paper or web based)
10
III. Method the CCQCST model
Sixth step 6. CORRECTION The standardized
test developer carries out the correction of the
tests. End products Treatments are designed
to check the quality of the questions before the
final version of the results
3. Questions
Selection or writing of the questions which are
linked to the trinomials selected
Item bank
4. Training
on the modalities of the questions defined
Blank tests (paper or web based)
5. Testing
The standardized test is proposed
Online exams, paper tests linked with OMR
11
III. Method the CCQCST model
Seventh step 7. FEEDBACK Personalized and
tailored feedbacks are proposed on the basis of
the final version of the results. End products
Personalized and tailored feedbacks with online
access.
3. Questions
Selection or writing of the questions which are
linked to the trinomials selected
Item bank
4. Training
on the modalities of the questions defined
Blank tests (paper or web based)
5. Testing
The standardized test is proposed
Online exams, paper tests linked with OMR
12
III. Method the CCQCST model
Eight step 8. REGULATION the test developer
collects and analyses the opinions of the all the
people who intervened in the test construction
process. End products Advices in order to
improve the next standardized test that will be
created.
3. Questions
Selection or writing of the questions which are
linked to the trinomials selected
Item bank
4. Training
on the modalities of the questions defined
Blank tests (paper or web based)
7. Feedback
5. Testing
The standardized test is proposed
Personalized and tailored feedbacks are proposed
on the basis of final version of the results
Online exams, paper tests linked with OMR
online access FB
13
III. Method the CCQCST model
14
III. Method the CCQCST model
15
V. Companies organizations collaborating in
this project
M. Bernard Van ParysM. Hubert de Broqueville
M. Francis LajoieM. Jean-Pierre Renault
M. Alain Kock
M. Dominique PierretM. Jean-Marie Pairoux
Mme Régine GendarmeMme Catherine DeruisseauMme
Rameault
Mme Sandra SchillemansMme Tania DelhaiseM.
Pierre Ramelot
Mme Maryse RiffontM. Peter Vanderheyden
Mme Catherine WarnantM. Jean-Philippe Léonard
M. Yvan Lagage
16
VI. First results Relevance of the CCQCST model
17
VI. First results Relevance of the CCQCST model

• Interviews of tests developers of the ten
  companies and organizations
• Description of the actions
• Collaborators concerned
• Temporal dimension of the tests construction
  process
• Other processes in which tests are inserted
• Importance of the quality of the standardized
  tests.
• First results of the interviews and the immediate
  subcontracting services
• The CCQCST model and its procedures are
  considered as relevant and well accepted
• The principal readjustment we have to plan
  consists of an integration of different roles of
  tests developers working in teams.

18
VI. First results Users roles within e-CQCST
platform

• Two difficulties for the users roles definition
• They must be used in very different companies or
  non academic organizations contexts
• They should be assigned by one person in charge
  of the test (the head of the project) to a series
  of collaborators or subcontractors.
• We propose seven roles within the e-CQCST
  platform
• Role 1 INSTALLER ? installs the software on
  the server,
• Role 2 ADMINISTRATOR ? defines the different
  tests supervisors,
• Role 3 TEST SUPERVISOR ? defines and
  supervises his collaborators
  subcontractors for the tasks related to
  the test construction,
• Role 4 TEST DEVELOPER ? realizes one or more
  tasks of one or more steps
  of the CCQCST model,
• Role 5 QUALITY SUPERVISOR ? defines and
  supervises his collaborators or
  subcontractors in charge of the
  quality controls,
• Role 6 QUALITY CONTROLLER ? realizes one or
  more tasks related to the
  quality controls of the productions of the
  tests developers,
• Role 7 ASSESSEE ? performs the standardized
  test.

19
VI. First results The SAFES project

• Being given the success of the subcontracting
  services requested by four companies, we filed in
  a FIRST SPIN-OFF project of the DGTRE.
• Our project entitled Smart Advices For
  Evaluation Systems (SAFES) was well ranked by an
  independent scientific committee so that we are
  currently, starting a two years feasibility study
  that will allows us
• To create a method of audit of standardized
  testing systems
• To improve the quality control procedures to be
  implemented
• To define a typology of subcontracted services
• To realize a market study.
• The creation of the spin-off company should
  coincide with the finalisation of the e-CQCST
  platform.

20
VI. First results First interfaces of the
e-CQCST platform

• General ergonomics

21
VI. First results First interfaces of the
e-CQCST platform

• Management of the e-CQCST users by the test
  supervisor

22
VI. First results First interfaces of the
e-CQCST platform

• 1st module ANALYSIS - specification table PE
  are crossed with CP

23
VII. Conclusions

• Docimological frameworkThe CCQCST model is
  relevant and well appreciated in the context of
  the ten companies and organisations which are our
  partners in this project
• Ergonomics aspectsThe first results show that
  the ergonomics of the first interfaces facilitate
  the production of specification tables related to
  the first step of the standardized tests
  construction
• Economical valorization frameworkThe surprising
  immediate docimological subcontracting services
  requested by four of the ten companies (also the
  acceptation of the SAFES project) shows that
  there are possibilities for economical
  valorisations of the results of this project.
• Users frameworkThe seven categories of users
  roles we defined were validated by our partners.
  This clarification should contribute to highlight
  the complementarities and the responsibilities of
  the persons who are involved in standardized test
  creation.

24
Thank you for your attention ! For more
information concerning the electronic
Construction and Quality Control in Standardized
Tests project (e-CQCST) do not hesitate to
contact me jl.gilles_at_ulg.ac.be http//www.smar
t.ulg.ac.be/earli-2005/e-CQCST.ppt