The%20training%20needs%20of%20different%20stakeholders%20in%20language%20testing%20and%20assessment

1
The training needsof different stakeholdersin
language testingand assessment

• Angela Hasselgreen, Ari Huhta, Jayanti Banerjee

2
Outline

• The design and implementation of the
  questionnaire
• Some overall themes
• Comparing results by regions
• Implications

3
The timetable

• First meeting
• Preliminary design
• Web-based questionnaire (roll-out date)
• Data collection
• Data preparation (for analysis)
• Analysis preparation for conference

• 24 25 January 2004
• 1 February 2004
• 1 March 2004
• 1 March 15 April 2004
• 18 21 April 2004
• 23 April - 12 May 2004

4
The instrument
5
(No Transcript)
6
Part 2 Questions for teacher trainers/educators P
art 3 Questions for people who work, full time
or part time, for language testing/examination
boards or organisations that design
school-external, regional or nationwide
tests/examinations
7
Summary of responses

• 914 respondents to at least one part of the
  questionnaire
• Part 1 741
• Part 2 709
• Part 3 689
• 501 respondents have provided contact details

8
Summary of responses

• Point to Note
• Respondents did not necessarily answer every item
  in each section for which they submitted data.
• Responses for parts 2 3 are particularly low
• Part 1 522 614 responses per item
• Part 2 181 212 responses per item
• Part 3 198 228 responses per item

9
Where do the respondents work?
10
Teachers only degree of formal training in
assessment
1 none 2 short 3 medium 4 long
11
  Correlation between assessment-related
activities teachers are engaged inand absence of
professional training 1 Classroom-focussed
assessment
12
Teachers perceived needs do you need training
in this area? 1 Classroom-focussed assessment
13
Correlation between assessment-related activities
teachers are engaged inand absence of
professional training2 Purposes of assessment
14
Teachers perceived needs do you need training
in this area? 2 Purposes of assessment
15
Correlation between assessment-related activities
teachers are engaged inand absence of
professional training3 Content and concepts
16
Teachers perceived needs do you need training
in this area? 3 Content and concepts
17
Correlation between assessment-related activities
teachers are engaged inand absence of
professional training4 External tests and exams
18
Teachers perceived needs do you need training
in this area? 4 External tests and exams
19
Tentative conclusions on teachers needs

• Teachers generally perceive a need for training
  across the board
• Areas of particular need
• ELP/other portfolios, preparing own tests,
  peer/self assessment, interpreting results and
  giving feedback
• To award grades, to find out what needs to be
  taught, placing students, awarding certificates
• Aspects of culture, integrated skills,
  establishing reliability and validity, statistics
• Reviewing and writing items, statistics, defining
  criteria, rating, interviewing

20
Teacher trainers what they train
1 Classroom-based
21
Correlation between assessment-related activities
teacher trainers give training in, and absence of
their own professional training
1 Classroom-based
22
Teacher trainers what they train2 Purposes
23
Correlation between assessment-related activities
teacher trainers give training inand absence of
their own professional training 2 Purposes
24
Teacher trainers what they train3 Contents and
concepts
25
Correlation between assessment-related activities
teacher trainers give training inand absence of
their own professional training 3 Contents and
concepts
26
Tentative conclusions from teacher trainer data

• Teacher trainers perceive their own needs
  similarly to teachers
• Much of what they train in is in areas in which
  they have no professional training themselves
  (generally 30-50)
• Areas of greatest need, where teacher training is
  not given, or given by people with little
  professional training, largely coincide with the
  areas of greatest need in teachers
• ELP/other portfolios, preparing own tests,
  peer/self assessment, interpreting results and
  giving feedback informal continuous assessment
• To award grades, to find out what needs to be
  taught, placement, awarding certificates
• Aspects of culture, integrated skills,
  establishing reliability and validity, statistics
  productive skills

27
Correlation between activities experts are
engaged in and absence of professional
training  1 Item writing and rating
28
Experts perceived needs do you need training
in this area? 1 Item writing and rating
29
Correlation between which activities experts are
engaged in and absence of professional
training.2 Developing tests and assessment
systems
30
Experts perceived needs do you need training
in this area? 2 Developing tests and
assessment systems
31
Correlation between which activities experts are
engaged in and absence of professional
training3 CEF related
32
Experts perceived needs do you need training
in this area? 3 CEF related
33
Tentative conclusions on experts needs

• Experts generally perceive need for training
  across the board
• Areas of particular need
• Item writing, reviewing items or tests, making
  decisions about composition of tests
• Statistical analysis, setting pass marks,
  creating/maintaining item banks, doing validation
  research
• Relating existing tests to the CEF, using CEF as
  basis for new test systems

34
Regions compared

• 1. Southern Europe
• Portugal, Spain, Andorra, Italy, Malta, Greece,
  Turkey
• 2. Western Europe
• Ireland, UK, France, Belgium, Netherlands,
  Germany, Austria, Switzerland
• 3. Northern Europe
• Iceland, Norway, Sweden, Denmark, Finland
• 4. Eastern Europe
• Estonia, Latvia, Lithuania, Poland, Russia, Czech
  Rep., Slovakia, Hungary, Bulgaria, Romania,
  Ukraine, Slovenia, Croatia, Serbia and
  Montenegro, Macedonia, Azerbaijan
• 5. non-European countries
• Questionnaire 1 / Teachers

35
Teachers previous training2 A little (1-2
days)
36
Teachers need for training- average number of
activities, etc where need for assessment
training was reported (out of max 25)
37
Teachers reported need for advanced training 1
(activities out of max 25)
38
Teachers reported need for advanced training 2
(activities out of max 25)
39
Statistically significant differences

• Teachers previous training
• no differences between regions
• Teachers need for advanced training
• Western Europe lt Eastern Europe
• Western Europe lt Southern Europe
• Teachers need for basic training
• no statistically significant differences
• small n size made it impossible to calculate
  differences between individual countries

40
Top 5 needs (teachers / regions)
1st 2nd 3rd 4th 5th
South Portfolio 73 Interpretation 65 Statistics ext. 64 Continuous 62 Criteria 61
West Reliability 55 Validity 54 Statistics 50 Portfolio 45 Self / peer 45
North Portfolio 70 Self / peer 63 Reliability 62 Validity 62 Statistics 61
East Portfolio 72 Reliability 72 Statistics 71 Validity 71 Interpretation 65
41
Implications for training event(s)

• Common training events could be designed for both
  teachers and teacher trainers
• These should aim to cover the needs revealed here
  as pressing however, it is likely that more
  core needs may exist for grass roots teachers
  (further investigation needed)
• A separate, specialist, training event would be
  necessary to cater for experts needs
• Regional events may be adapted to reflect
  different priorities

42
And finally..Beneficial to belong to EALTA?