ICAO-Special Workshop on Language Proficiency, - PowerPoint PPT Presentation

Loading...

PPT – ICAO-Special Workshop on Language Proficiency, PowerPoint presentation | free to view - id: 729da8-ZTBlM



Loading


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation
Title:

ICAO-Special Workshop on Language Proficiency,

Description:

Standards for selection and development of language tests Dr Jeremy Mell Head of Language Studies ENAC, Toulouse, France jeremy.mell_at_enac.fr – PowerPoint PPT presentation

Number of Views:76
Avg rating:3.0/5.0
Slides: 28
Provided by: MELL173
Category:

less

Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: ICAO-Special Workshop on Language Proficiency,


1
Standards for selection and development of
language tests
  • Dr Jeremy Mell
  • Head of Language Studies
  • ENAC, Toulouse, France
  • jeremy.mell_at_enac.fr

2
Options for assessing language proficiency for
aviation
  • informal observations
  •  line checks ,  inspections 
  • classroom assessments
  • formal language tests
  • direct/semi-direct/indirect
  • simulated language use
  • most direct
  • integrative performance samples are matched to
    rating scales
  • paper and pencil OR screen and mouse tasks
  • most indirect
  • discrete items/skills favour numerical scores
  • but  juggling with only one ball .

3
Language tests
  • Definition
  • structured events or procedures
  • to elicit performances as samples of
    test-taker s language skills
  • in a standardised way
  • enabling reliable inferences to be made
    concerning his/her level of competence
  • and possibility of reproducing those skills at
    that level of competence consistently over time
  • adapted from Carrol (1968) and Douglas (2000)

4
Language testing
  • A well-developed domain of intellectual activity
  • academic
  • body of research, cross-disciplinary links,
    international associations,...
  • codes of ethics, codes of practice
  • institutions of excellence,  chapels ,
     gurus , feuds
  • commercial
  • international testing service providers
  • general, academic, business language varieties
  • Attempts to link different rating scales/test
    results

5
Language test qualities
  • general qualities
  • validity
  • face validity
  • reliability
  • positive washback
  • practicality
  • security
  • conflicting qualities
  • developement procedures
  • needs analysis (language corpora, surveys,.)
  • design/item-writing team (operational,
    linguistic, testing expertise)
  • trialling (representative populations, controlled
    conditions, data retrieval)
  • statistical tools
  • test documentation
  • examiner training, accreditation

6
Types of language tests
  • entry
  • placement
  • diagnostic
  • progress/achievement
  • proficiency

7
Entry/placement tests
  • Purposes
  • ab initio training
  • recruitment
  • form level groups
  • recurrent training
  • benchmarking a population
  • form level groups
  • Characteristics
  • items chosen to cover a broad range of levels of
    general language
  • formats reflect previous education
  • scores tend to form a  bell curve 
  • stakes
  • high (recruitment)
  • mid (benchmarking)

8
Diagnostic tests
  • Purposes
  • identify specific areas of skill/knowledge for
    improvement in subsequent training
  • individualisation of training programmes
  • Characteristics
  • each item chosen to represent a single
    significant area of knowledge/skill
  • overall score less important than analysis of
    right/wrong responses - focus on errors
  • stakes low

9
Progress/achievement tests
  • Purposes
  • to measure effectiveness of a phase of learning
  • to allow access by learner to next phase of
    learning
  • Characteristics
  • items chosen to closely reflect content and
    methodology of preceding training phase
  • scores will often be interpreted with regard to
    average or pre-established norm
  • may refer to intermediate levels (3,5 4,5)
  • stakes low to medium

10
Proficiency tests
  • Purposes
  • to establish the competence of candidate to
    exercise language skills in operational
    conditions
  • Characteristics
  • items chosen to resemble real-world tasks
  • overall scores are holistic YES/NO
  • stakes very high
  • a blunt instrument but must be highly robust
  • proven validity
  • proven reliability
  • security

11
Testing needs of aviation
  • Ab initio populations
  • entry
  • selection
  • local education vs aviation training policies
  • progress/achievement
  • intermediate training objectives
  • continuation/curtailment
  • initial proficiency
  • licensing
  • Qualified populations
  • placement/diagnostic
  • benchmarking populations
  • individual training needs
  • acceptance by takers
  • (progress/achievement)
  • degree and speed
  • proficiency renewal
  • recurrent

12
Test design specifications (ICAO)
  • As of 5 March 2008, aeroplane and helicopter
    pilots, air traffic controllers and aeronautical
    station operators shall demonstrate the ability
    to speak and understand the language used for
    radiotelephony communications to the level
    specified in the language proficiency
    requirements in the Appendix. (Annex 1, 1.2.9.4)
  • FORMAL EVALUATION
  • An individual must demonstrate proficiency at
    level 4 in all categories in order to receive a
    level 4 score. (Doc 9835, Manual on the
    Implementation of ICAO LPRs, 2.8.4)
  • PROFILE EVALUATION

13
Institutional context
  • possibility of independent international
    oversight to ensure
  • validity of testing procedures used
  • comparability of testing procedures and outcomes
  • testing outcomes are linked with personnel
    licensing
  • testing may not be in the sole hands of the
    language community
  • partnerships with CAA, service provider,
    airline,...
  • system cannot tolerate a high failure rate
  • waste of expensive training investment
  • staffing levels must ensure continuity of service
  • prior knowledge of test procedures for positive
    washback on training
  • trainee motivation and attitude, reassurance of
    candidates
  • development of appropriate training systems
  • BUT unsuitability of cramming, test item
    banks,...

14
A common test for ATC and pilots?
  • pilots and controllers are partners in R/T
    communication
  • but they are set apart by
  • different interactive roles
  • complementary passive/active competencies
  • opportunity to use other job-related language
    uses to extend speech sample
  • controller telephone co-ordinations, report to
    supervisor, ...
  • pilot pre-flight, intra-cockpit, cabin
    announcements, ground staff, .
  • a possible solution?
  • common core tests of knowledge
  • job-specific components for testing competence

15
Need for well-planned testing services
  • time/space constraints
  • dispersed locations (recurrent)
  • irregular schedules and limited availability of
    test-takers (recurrent)
  • availability of technologies
  • test/re-test
  • test components
  • single event, integrative?
  • different item designs to test separate skills?
  • standardisation of results
  • extensive trialling
  • comparable conditions of test administration
  • examiner training/auditing based on speech
    samples
  • interlocutors
  • raters
  • paired/multiple rating
  • test security
  • secure storage and transfer of test materials
  • multiple parallel versions

16
Appropriate technologies
  • voice-only
  • telephone, 2-way radio, language laboratories,
    training simulators,...
  • input delivery
  • analogue players, computer screen (multimedia)
  • performance storage and access
  • analogue recorders, computer sound files (rapid
    access)
  • speech recognition
  • standardise perception of intelligibility, save
    rating costs
  • BUT needs to filter all possible speech
    variations
  •  interactions  need to be human-human
  • initiate and maintain exchanges
  • deal with misunderstandings

17
Rater/interlocutor qualifications
  • 3 basic qualifications
  • language level
  • level 6 (language raters)
  • level 5/6 (interlocutors, SME raters)
  • aviation background familiarity with ICAO LPRs
  • principles of language proficiency and language
    testing
  • suitable candidates
  • aviation personnel
  • language trainers
  • L1 background
  • native speakers(NS)/non-native speakers (NNS)
  • risk of familiarity with a given form of NNS
    spoken English

18
Assessment of level 6
  • NOT a safety issue, but raises concerns about
  • resources ( no re-test)
  • NS/NNS distinctions
  • Test-takers may be
  • mother-tongue
  • non-mother tongue
  • intranational use
  • non-intranational use
  • Candidacy on basis of biographical criteria
  • citizenship, educational background, residency,
    ...
  • Protocols for assessment
  • admissible evidence
  • assessor qualification
  • appropriate documentation
  • Subsequent checking and oversight must address
  • adherence to standard ICAO phraseologies
  • ICAO LPRs for
  • intelligibility
  • appropriateness

19
Points of view within the aviation English
profession
  •  We need to find workable and practical
    solutions to the training and testing challenges
    of the next few years. Those solutions may not be
    able to wait for the  perfect  aviation English
    proficiency test.
  • ICAEA_WORLD discussion group, 29th May 2004.
  • I have some considerable concerns about an
    individualistic approach being adopted by
    airlines / ATC administrations / civil aviation
    authorities around the world. my view is that
    it is essential that testing be subject to
    external standards and be standardised and
    universal to some degree.
  • ICAEA_WORLD discussion group, 14th April 2004

20
Informal evaluations
  • Within the initial or continuing training
    process, language trainers can identify 3
    categories of trainee
  • well-below level 4
  • approximately at level 4 (confirm by formal
    evaluation)
  • comfortably above above level 4
  • thus enabling informed decisions to be made on
  • possible access to professional functions
  • further training required
  • benchmarking of populations

21
Equivalences?
  • ICAO level 4 is specific to proficiency
    requirements of aeronautical communications
  • There are no validated equivalences with bands on
    other rating scales or scores in general purpose
    tests...
  • .. however scales/scores relating to listening
    and speaking could be used to identify  optimal
    points of transition  to ICAO LPRs for selection
    purposes, eg
  • IELTS test bands 5/6
  • Common European Framework levels B2/C1
  • Cambridge Testing Suite FCE (high pass)/CAE (low
    pass)

22
Formal tests for ICAO level 4
  • Placement (entry)
  • must relate to ALL 6 skills in ICAO scale
  • must measure distances above and below level 4
  • may have diagnostic capability
  • tasks/topics/input materials focus on general or
    job-related skills
  • development requires some statistical validation
  • Proficiency (exit)
  • must relate to ALL 6 skills in ICAO scale
  • must characterise test-takers as BELOW or AT
    levels 4, 5 or 6
  • tasks/topics/input material must reflect
    radiotelephony language competencies (including
    standardised phraseology)
  • development requires strong statistical
    validation (high stakes)

23
Detailed specifications (proficiency)
  • provide a representative range of intelligible
    international accents as input for comprehension
  • provide a professionally relevant format for
    candidates to display comprehension
  • elicit an adequate continuous speech sample to
    test fluency/pronunciation
  • provide a voice-only setting for diadic
    (2-person) interactions
  • provide examples of routine and unexpected events
    in a work-related context
  • allow the candidate to use basic grammatical
    structures creatively
  • allow the candidate to demonstrate ability to
    paraphrase
  • allow the candidate to change between
    rehearsed/formulaic speech and spontaneous
    interaction
  • simulate unexpected events to create
    opportunities for misunderstanding.
  • EUROCONTROL/ENAC preliminary feasibility study
    2004

24
Some existing specific tests
  • Controllers
  • ab initio
  • entry EPT/FEAST(Eurocontrol)
  • progress APRO (ENAC, F)
  • proficiency PELA (Eurocontrol)
  • recurrent
  • placement TNP (DGAC, F)
  • proficiency ELPAC (Eurocontrol)
  • under development for 2007
  • Pilots
  • ab initio
  • proficiency FCL 1.200 (DGAC, F)
  • Other tests
  • TOEFA (Peru),
  • TELPA (IAES, Korea)
  • TELAP (CA Flight University,China)
  • RELTA (RMIT, Australia)
  • TEA (Mayflower, UK)
  • .

25
TNP (DGAC, France)
  • developed by national language and subject-matter
    experts (controllers and ATC management) as a
    placement/diagnostic tool
  • 3 papers cover professional language content
  • a written multiple choice test of language
    knowledge
  • a tape-mediated written test of listening
    comprehension (RT messages)
  • an individual oral interview using paper-mediated
    graphic and text input
  • expresses results in terms of the ICAO level
    chart
  • used since 1999 with qualified air traffic
    controllers
  • benchmarking to establish national training needs
  • formulating periodic individualised language
    development and maintenance programmes in order
    to meet national qualification renewal
    requirements

26
PELA (Eurocontrol)
  • developed as a proficiency testing tool as part
    of European ATM harmonisation
  • 3 papers cover professional language content
  • a tape-mediated written test of listening
    comprehension (RT and telephone co-ordination
    exchanges)
  • a tape-mediated test of interaction (short
    responses) in routine ATC situations
  • a live interaction with trained interlocutor
    simulating a non-standard ATC situation and
    report to supervisor
  • used in Eurocontrol/ECAC states with ab initio
    controller trainees at end of basic training
  • recently revised by language and subject-matter
    experts to conform with ICAO language proficiency
    requirements
  • computer-based versions of Papers 1 and 2
    currently under development

27
Further reading
  • Bachman, L. and Palmer, A. 1996. Language Testing
    in Practice. Oxford Oxford University Press.
  • Carroll, J. B. (1968). The psychology of language
    testing. In A. Davies (Ed.), Language Testing
    Symposium A Psycholinguistic Perspective (pp.
    46-69). Oxford Oxford University Press.
  • Douglas, D. 2000. Assessing Languages for
    Specific Purposes. Cambridge Cambridge
    University Press.
About PowerShow.com