Title: Linking exams to the CEF levels: The Manual and the Catalan experience
1Linking exams to the CEF levelsThe Manual and
the Catalan experience
- Neus Figueras
- nfiguera_at_xtec.net
2Presentation Outline
- Background information about the Catalan EOI
- Usefulness and relevance of the CEF and the
Manual - The project to link our exams to the CEF
- Using The Manual problems encountered and
solutions adopted - A Manual for the future
3Background
4The EOI system in Catalonia
- State funded language schools (16)
- Two levels defined
- Certificat de Cicle Elemental (aimed at B1)
- Certificat dAptitud (aimed at B2)
- 13 different languages (7 6)
- 15,000 thousand certificate exams every year
- Standardised language proficiency certificates in
place since 1995, with exams - developed centrally
- related to published curricula
- written according to set specifications
- submitted to pre-testing
- standardised administration and marking
- analysed statistically
5Issues to be solved in 2002
- Revise EOI certificate exams after 10 years.
- Develop an item banked system.
- Link levels to Common European Framework levels.
-
-
-
- Is examination difficulty equivalent across time?
- Is the lower certificate consistently easier than
the higher certificate? - Are the different certificates in the different
languages comparable?
6To sum up
Teaching/learning context
7Why linking?
8Why the CEF?
9A link or a knot?
10 Linking or equating?
A link or a knot?
Adoption or adaptation?
11Relevance of the Manual (2003)
- Clarified issues tests and the CEF levels
- Provided set of systematic procedures
- Stated what was right
- Followed CEF approach and philosophy
- Self-justified claims of linkage to the CEF
12The Manual offers guidance to users to...
- describe the examination coverage, administration
and analysis procedures, - relate results reported from the examination to
the Common Reference Levels presented in
Chapter 3 of the CEF, - provide supporting evidence that reports the
procedures followed to do so. -
13The Manual steps
- Four inter-related sets of procedures which
- contribute to the validation process
- Familiarisation
- Specification
- Standardisation
- Empirical Validation
14(No Transcript)
15 16 Prerequisites for linkage
- If an examination result is not reliable, then
the examination cannot be compared to the CEF. - If each time a new form of an examination is
produced it varies according to content and
difficulty, it is very difficult to compare the
examination to the CEF since the examination does
not present a stable standard. - If examination results vary by teacher and
professor, by school or university, they cannot
be matched to the CEF. -
- (Alderson 2002)
17(No Transcript)
18Our case.
Teaching/learning context
19Our case project Outline 2003-06
- July 2003 2004
- Define objectives
- Project design
- Empirical scale development
- Scale validation
- Item banking (English) starts
- July 2004 2005
- Publication of scales
- Item banking (English) continues
- Linkage to CEF Procedures in the Manual
(specification) - July 2005 2006
- Linkage to the CEFProcedures in the Manual
(standardisation) - Linkage to the CEF
(empirical validation) - Comparability across languages
20Scale Development....à la mode du CEF
- Step 1 selecting level descriptors (per skill)
- CEF
- Curriculum objectives for EOI
- Finnish scales (YKI)
- Step 2 translation of the descriptors into
Catalan - Step 3 mapping descriptors onto levels (CEF and
EOI) - Experts assign levels to descriptors
- Analysis of sorting exercises
- Collating resulting surviving descriptors
- Step 4 developing new scales
- Drafting
- Validation
- Scale revision
- Validation
- Final version
21Linking CEF and EOI scales (www.xtec.net/eoi)
22Comparing EOI and CEF levels (Voc)
- CEF - C1 Has a good command of a broad lexical
repertoire allowing gaps to be readily overcome
with circumlocutions little obvious searching
for expressions or avoidance strategies. Good
command of idiomatic expressions and
colloquialisms. - EOI C1 Can understand and use a broad lexical
repertoire, including synonyms and antonyms of
less common words, idiomatic expressions, to
express himself/herself correctly and
appropriately on a wide range of general,
academic, professional and leisure topics. Can
understand word play based on multiple meanings
and use prefixes and suffixes to form less common
words. Less frequent words may cause some effort
in comprehension.
23Linking CEF and EOI scales
24Lessons learntimplications
- The CEF is not always right
- The CEF as a catalyst
- Reading is not understanding
- Continued training is necessary
- Appearances need to be checked empirically
- Exact correspondence may not be possible, but is
it desirable? - Linkage does not mean equivalence
25Item bank building (first batch, June 2004)
- Step 1
- Booklet development (Listening, Reading, Grammar
Vocabulary). (Total items included 396) - Step 2
- Data collection (260-784 students per item)
- Analyses CTT and IRT
- (Total surviving items 301)
- Step 3
- Standard setting procedures
- - test centered
- - examinee centered
- - annual average pass rate
- Setting (provisional) cut off scores at Elemental
(B1) and Aptitud (B2)
26And nowback to the Manual
27Problems encountered Specification
- The gaps and inconsistencies in the CEF are
evidenced - List of communication themes clearly not
definitive ( p.52) - Definition of communicative task/communicative
activity (4.3, 4.4.2.2) - Definition of tasks (7.1, 7.2, 7.3)
- Grids useful as awareness raising but also
frustrating - On what basis are the grids completed?(Testversion
/Specs?) - Who does what?
- Who checks it?
28Solutions adopted Specification
- Three languages
- Team work in 5 steps (for each grid)
- Judges go and have a first attempt to fill up
- Joint work in understanding/agreeing on
interpretation for each cell in each grid - Judges go away and do the job
- Judges meet again and discuss possible problems
- External judge checks three languages for
homogeneity. - How ?
- Extensive knowledge of CEF and Manual
- Insiders information from the Dutch CEF
Construct project - No commercial (or other) interest in confirming
our linkage - No hurry to be on the CEF claimers map
29Problems encountered Standardisation
- Reference materials not ready until late 2005.
- Different skills, different languages, different
nature. - reference with a small r (quantity and
quality) - Assessing task vs. item levels.
- No linguistic competence reference materials.
30Solutions adopted Standardisation
- Taking advantage of judges/system used to
standardisation procedures in speaking and
writing. - Using part of familiarisation process together
with exam-centered standard setting procedures.
31Empirical validation Problems encountered,
solutions still
under way.
- Internal validation back to
- External validation
- Anchoring and linkage with small r
materials - Overcoming difficulties with design in
languages with small number of
candidates.
32A straightforward alternative
33The proposal from Dutch CEF project
- Train in CEF labelling instruments.
- Describe the test (texts and items) using the
dimensions in the Final Grid. - Make a guess at the level of an item (guided by
the classification system), leading to the
estimated CEF level. - Pretest the items thus labelled.
- Calibrate the items.
- Do standard-setting to set the boundaries of the
levels on the scale coming from the calibration. - Assign a psychometric level to the items.
- Assign a definitive level to the items.
34but beware.
- An item can only be assigned to a definitive
level - if the psychometric level falls within the band
of - the estimated level (in other words if the
estimation - based on the content is comparable with the
- psychometric value)
- ?
35And mind the gaps!
- Coverage in terms of CEF relevant subscales
- Comparison across countries, institutions,
education levelsand viceversa - Comparison across skills and languages
36The Manual for the future
- Whats it going to be like? Structure and Range
- What currency?
- Whos going to use it?
- How (far)?
- Will there be primus inter pares?
37(No Transcript)
38-
- O mon âme, naspire pas à la vie immortelle,
mais épuise le champ du possible. - Pindare, 3e Pythique
-
39Available tools so far.
- Dutch CEF construct Project www.ling.lancs.ac.uk/c
efgrid - Council of Europe Pilot samples www.coe.int
- English speaking performances
- French speaking performances
- English, German, Spanish, Italian
listening and reading tasks - (Available from Johanna.PANTHIER_at_coe.int )
- ALTE Writing Grid
- ALTE Speaking Grid
- - CEFTRAIN www.ceftrain.net