Briefing%20Session%20on%20the%202012%20HKDSE%20English%20Language%20Assessment%20Framework

1
Briefing Session on the 2012 HKDSE English
LanguageAssessment Framework

• March 2008

2
Background

• Third consultation conducted in late 2006
• New public assessment of HKCEE English Language
  first implemented in 2007
• Further consultation in the light of the 2007
  HKCEE experience
• Questionnaire survey conducted in January 2008

3
2007 HKCEE Experience
4
SBA Implementation Schedule
Year Options for School
2007 1. Submit SBA marks for feedback marks included as 15 of the subject result 2. Submit SBA marks for feedback only marks not included in subject result 3. Not submit SBA marks exam results to contribute 100 of subject result
2008 1. Submit SBA marks for feedback marks included as 15 of the subject result 2. Submit SBA marks for feedback only marks not included in subject result
2009 1. Submit SBA marks for feedback marks included as 15 of the subject result
5
Implementation of SBA in 2007
No. of schools Percentage () No. of candidates Percentage ()
Option 1 (Yes) 199 34 31,876 43
Option 2 (Trial) 125 22 20,945 28
Option 3 (No) 254 44 21,388 29
Total 578 100 74,209 100
6
Moderation Results of Option 1 Schools
The mean of the SBA scores is No. of Schools Percentage ()
within the range 144 72.4
slightly higher than expected 29 14.6
higher than expected 2 1.0
much higher than expected 0 0
slightly lower than expected 21 10.6
lower than expected 3 1.5
much lower than expected 0 0
7
Moderation Results of Option 1 Schools
The standard deviation of the SBA scores is No of Schools Percentage ()
as expected 179 89.9
slightly wider than expected 0 0
wider than expected 0 0
slightly narrower than expected 10 5.9
narrower than expected 9 4.5
much wider than expected 0 0
much narrower than expected 1 0.5
8
Moderation Effect on Candidates
Mark Adjustment ( of subject) No. of candidates Percentage of candidates ()
0 (0) 5365 17
1-3 (lt1) 19881 62
4-6 (lt2) 6237 20
7-9 (lt3) 392 1
31,875 100
9
Implication of Moderation Results

• Mean and spread of the SBA marks submitted by the
  majority of schools fall within the expected
  range
• Most teachers have a good understanding of the
  assessment criteria and can assess their students
  accurately
• Reliability of SBA not an issue
• Fairness ensured by statistical moderation

10
Absentee Rate of School Candidates in 1996 CE
English Language Speaking Exam
English Language (Syllabus A) No. Sat 6,623
English Language (Syllabus A) No. of Absentees 3,058
English Language (Syllabus A) Absentee Rate 31.6
English Language (Syllabus B) No. Sat 88,201
English Language (Syllabus B) No. of Absentees 19,236
English Language (Syllabus B) Absentee Rate 17.9
11
Absentee Rate of School Candidates in 2007 CE
English Language Speaking Exam
No. Sat No. of Absentees Absentee Rate ()
Option 1 (Yes) 27,935 3,398 10.8
Option 2 (Trial) 19,307 1,466 7.1
Options 1 2 (Yes Trial) 47,242 4,864 9.3
Option 3 (No) 19,298 3,681 16.0
All Schools 66,540 8,545 11.4
12
Possible Explanations

1. Students in Yes and Trial schools had more
   speaking practice in school because of the SBA
2. They became more confident with the public exam
   format, which involves similar speaking tasks
   group discussion and individual response
3. Positive backwash effect of the SBA on the
   ability to speak English and to interact with
   peers in different contexts.

13
Effect of SBA on Speaking Exam
No. Sat Speaking Exam Mean Moderated SBA Mean
Option 1 (Yes) 27,804 25.38 27.81
Option 2 (Trial) 19,381 26.00 -
Option 3 (No) 21,293 23.51 -
All Schools 68,478 24.97 -
14
Effect of SBA on Speaking Exam

• Students in Yes schools got higher marks in the
  SBA than in the speaking exam
• Students in Yes and Trial schools got higher
  marks in the public speaking exam than those in
  No schools
• BUT Yes and Trial schools may just be better
  schools

15
Effect of SBA on Speaking Exam
No. Sat Actual Predicted Mean Scores t-value
Option 1 (Yes) 27,804 0.25 -1.97
Option 2 (Trial) 19,381 0.03 -0.21
Option 3 (No) 21,293 - 0.35 2.34
All Schools 68,478 - -
16
Effect of SBA on Speaking Exam

• Students in Yes schools did better than
  expected based on prediction from written papers
  (0.25)
• Students in Trial schools did slightly better
  than expected (0.03)
• Students in No schools did significantly worse
  than expected in the speaking exam (-0.35)

17
Congeneric Measures Model Analysis

• Strong evidence that the various components are
  measuring a single underlying ability
• Reliability of components .882 to .702
• R gt LIS gt W gt SBA gt S
• SBA (.753) more reliable than speaking exam
  (.702)

18
Inter-paper correlationsCorrelation
Matrix (by listwise case exclusion, N28,253)
R W L IS S SBA
R 1.000 .858 .887 .776 .803
W .858 1.000 .852 .767 .797
L IS .887 .852 1.000 .764 .796
S .776 .767 .764 1.000 .787
SBA .803 .797 .796 .787 1.000
19
Survey Results
20
Main focus of questionnaire

• Assessment of the modules in the Elective Part
• Replacing the speaking exam with SBA
• The need for strategic implementation of SBA in
  2012 HKDSE
• Teachers satisfaction with the level of support
  provided by the HKEAA and the EDB

21
Results Response Rate
All Schools Part. Schools Option 1 Schools Option 2 Schools Option 3 Schools
717 640 199 125 254
422 411 165 109 130
59 64 83 87 51
22
Results

• Responses measured on 6-point scale
• 3.5 adopted as minimum level of acceptability

23
Assessment of the Modules in the Elective Part
Question All schools Option 1 schools Option 2 schools Option 3 schools
1. Assessing the modules in the Elective Part of the curriculum through the writing paper is appropriate. 3.7 3.7 3.7 3.7
2. The proposed Paper 2 structure is appropriate. 3.8 3.8 3.8 3.7
3. The proposed weighting of Part A (10) and Part B (20) is appropriate. 4.0 4.0 4.0 4.0
4. The question types for the modules in the Elective Part are appropriate. 3.7 3.7 3.7 3.7
5. Assessing the modules in the Elective Part of the curriculum through Part B of the SBA is appropriate. 3.9 4.0 4.0 3.7
24
Assessment of the Modules in the Elective Part
Question All schools Option 1 schools Option 2 schools Option 3 schools
6. The proposed weighting of Part B (5) is appropriate. 3.6 3.7 3.6 3.5
7. The amount and type of effort required of teachers to implement the SBA in HKDSE is reasonable. 2.9 3.1 3.0 2.7
8. The amount and type of effort required of students to complete the SBA in HKDSE is reasonable. 3.4 3.5 3.5 3.3
9. Students have benefited from the introduction of SBA. 3.9 4.1 3.9 3.6
10. The effort spent on implementing SBA has been worthwhile. 3.5 3.7 3.5 3.3
25
Replacing the Speaking Exam with SBA
Question All schools Option 1 schools Option 2 schools Option 3 schools
12. School candidates need not take the public speaking examination as their speaking skills can be assessed by means of school-based assessment. 2.7 2.7 2.6 2.7
26
Teachers satisfaction with level of support
provided by HKEAA and EDB (1)
13. Resources and support provided by the HKEAA All schools
01. Handbook provided for the HKCEE English Language School-based Component 4.3
02. 2007 HKCE English Language Examination Introduction to the School-based Assessment Component (with DVD) 4.3
03. Training CDRs 1 and 2 4.1
04. HKCEE Chinese Language and English Language School-based Assessment (DVD distributed to parents and schools in October 2006) 3.8
05. HKCEE Chinese Language and English Language Information for Parents on School-based Assessment (leaflet distributed to parents in October 2006) 3.7
06. Statistical moderation of School-based Assessment scores (booklet distributed to schools in June 2007) 3.8
07. Group Coordinators System 3.8
27
Teachers satisfaction with level of support
provided by HKEAA and EDB (2)
08. Teachers seminars on the implementation of HKCEE English Language SBA component held in September/October 3.9
09. Assessment training sessions held in November (chaired by SBA group coordinators) 4.0
10. Inter-school sharing sessions (chaired by SBA group coordinators) 3.8
11. Professional development courses (conducted by HKU and HKIED) 3.9
12. SBA Corner on the HKEAA website (including General Information, List of Recommended Texts and FAQs) 4.0
13. The Platform for SBA of CE English Language Exam on the English Campus of HKedCity 4.3
14. The HKU website on SBA Projects 3.9
28
Teachers satisfaction with level of support
provided by HKEAA and EDB (3)
14. Resources and support provided by the EDB All schools
01. Resource materials 4.0
02. Professional development programmes 4.0
03. Grants 4.3
04. School-based Support Services 3.8
05. Collaborative research and and development (seed) projects 3.5
29
Need for Strategic Implementation of SBA in 2012
HKDSE
Question All schools Option 1 schools Option 2 schools Option 3 schools
16. My schools preparation plan for implementing the SBA for English Language in 2012 HKDSE is on track. 3.5 3.7 3.7 3.3
17. The proposed SBA framework for English Language is desirable for implementation in the 2012 HKDSE. 3.6 3.7 3.7 3.5
30
Respondents comments (1)
6. The proposed weighting of Part B (5) is appropriate. Part B should have a heavier weighting. Both Parts A and B should each account for 10 Both Parts A and B should each account for 5
31
Respondents comments (2)
7. The amount and type of effort required of teachers to implement SBA in the HKDSE is reasonable. Workload for teachers too heavy Too many assessments to do More resources needed, especially technical support More sample tasks provided
32
Respondents comments (3)
8. The amount and type of effort required of students to complete SBA in the HKDSE is reasonable. Reading six texts in three years and having to do four assessments in two years is too demanding. Hard to monitor time spent by candidates on SBA reading/viewing programme
33
Respondents comments (4)
18. State the kind of modification(s) to the English Language framework you think would be desirable for the 2012 HKDSE. Provide sample tasks Improve standardization SBA weighting should be increased/reduced Use outside examiners for SBA Keep Speaking exam Reduce the number of assessments
34
Proposed/Revised Assessment Frameworks
Current proposed framework Current proposed framework Revised proposed framework Revised proposed framework
Components Weighting Requirements Weighting Requirements
Paper 1 Reading 20 20
Paper 2 Writing 30 25
Part A 10 Word limit 200 10 Word limit 200
Part B 20 Word limit 400 15 Word limit 400
35
Proposed/Revised Assessment Frameworks
Components Weighting Requirements Weighting Requirements
Paper 3 Listening Integrated Skills 30 30
Paper 4 Speaking 20 For private candidates only 10 For all candidates
36
Proposed/Revised Assessment Frameworks
Components Weighting Requirements Weighting Requirements
SBA 20 For school candidates only 15 For school candidates only
Part A 15 Three assessments group interaction and individual presentation report three marks 10 Two assessments group interaction and individual presentation report two marks
Part B 5 One assessment based on the modules in the Elective Part of the curriculum report one mark 5 One assessment based on the modules in the Elective Part of the curriculum report one mark
37
Rationale for the Revised Framework

• Teachers need to deal with HKCEE, HKALE and HKDSE
  from 2009 to 2012
• Speaking exam retained for school candidates as
  the majority of teachers do not favour its
  replacement by SBA
• Weighting of SBA component reduced from 20 to
  15 and weighting of Part B of Writing paper
  reduced from 20 to 15 so that 10 can be
  allocated to the Speaking exam
• The total number of SBA marks to be submitted
  reduced from 4 to 3 to address teachers concerns
  about workload.
• Weighting of SBA component reduced to 15, which
  is the same as the 2007 HKCEE

38
Schedule for HKDSE Development

• Standards-referenced Reporting (SRR)
• Production of sample papers and marking schemes
• Pilot test of sample papers conducted
• Draft of level descriptors and annotated examples
  completed
• Briefing sessions conducted for schools SRR
  requirements finalised by Dec 08
• SRR booklets with sample papers disseminated to
  schools by Mar 09

39
Schedule for HKDSE Development

• School-based Assessment (SBA)
• Assessment criteria, specifications and sample
  tasks developed
• Briefing sessions conducted for schools SBA
  requirements finalised by Dec 08
• SBA handbook disseminated to schools by May 09

40

• Thank You
• Questions?