Changing Tests for Teachers: English Language School-Based Assessment in Hong Kong

1
Changing Tests for Teachers English Language
School-Based Assessment in Hong Kong

University of Hong Kong Faculty of Education
Inaugural Seminar Series English Language
School-Based Assessment Integrating Theory and
Practice, Dec 19th 2005

• Chris Davison
• Faculty of Education, University of Hong Kong

2
Curriculum and assessment reform in Hong Kong

• The Hong Kong Examinations and Assessment
  Authority (HKEAA) has recently moved from
  norm-referenced to standards-referenced
  assessment, including the incorporation of a
  substantial school-based summative oral
  assessment component into the compulsory English
  language subject in the Hong Kong Certificate of
  Education Examination (HKCEE).

3
Curriculum and assessment reform in Hong Kong

• The initiative aims
• to align assessment more closely with current
  English Language teaching syllabus (Curriculum
  Development Council, 1999) and the new
  outcomes-based Senior Secondary curriculum.
• to assess learners achievement in areas not
  easily assessed by public examinations, in
  particular speaking and extensive reading.
• enhance student self-evaluation and life-long
  learning.

4
The SBA initiative

• Starting in S.4, teachers assess their own
  students oral English language competencies
  through a range of authentic classroom-embedded
  activities over 2 years.
• Initiative developed by a team of researchers at
  the Faculty of Education, University of Hong
  Kong, in collaboration with HKEAA.
• Part of an evolving but coherent and systematic
  programme of research and evaluation,
  professional development and system-wide support

5
The SBA initiative

• SBA is integrated into the teaching and learning
  process, with teachers involved at all stages of
  the assessment cycle
• ? planning the assessment programme
• ? identifying and developing appropriate
  assessment tasks
• ? overseeing the assessment process
• ? making final judgments

6
The SBA initiative

• But previous studies have found
• Changes to summative assessment did not
  automatically lead to improvement in learning
  (Andrews 1994 Cheng 1998 Andrews, Fullilove
  Wong 2002).
• Assessment innovation was severely constrained by
  traditional school culture and by teacher,
  parental and student expectations (Cheung Ng
  2000 Carless 2001 Adamson Davison 2003).
• Wide variation in teachers interpretations of
  student performance and of their role in the
  assessment process (Yung 2001).

7
The SBA content and structure

• Aims
• Content
• Structure
• Assessment criteria
• Making judgments
• Standardization
• Safeguards

8
SBA as research

• The SBA initiative has stimulated multi-level and
  multi-purpose research efforts from a growing
  research team with more than 12 interrelated
  research projects now underway, or under review.
• In addition, the research efforts have attracted
  the interest of a growing number of MPhil and PhD
  students, as well as the attention of the
  international assessment and English language
  education research community.
•  
•  
•  
•  
•  
•  

9
SBA as research

• The particular characteristics of this research
  that makes it so interesting
• its significance to the educational community
• its high-impact and visibility
• its concern to address both fundamental
  theoretical problems in language assessment
  (basic research) and the needs of school
  communities (applied research)
• its action-oriented collaborative approach
•  
•  
•  
•  
•  
•  

10
SBA as research

• Many concerns and issues about SBA have been
  systematically gathered from key stakeholder
  groups during the process of SBA development,
  implementation and evaluation.
• Data was gathered over a year via questionnaires,
  individual and focus interviews, classroom
  observation and stimulated recall, and public
  briefing sessions as well as from the trialling
  of the tasks and assessment processes, and the
  training workshops, in total involving more than
  1800 teachers and over 600 schools.
•  
•  
•  
•  
•  
•  

11
SBA as research

• Adapting Brindleys (1995) taxonomy, these
• concerns and issues can be divided into three
• types
• Sociocultural (political)
• Theoretical (technical)
• Practical

12
Socio-cultural issues

• Still a traditional testing culture in Hong Kong
  
• Assessment practices are still primarily oriented
  towards providing data to select students for
  education or employment (Biggs 1995).
• External exam results are still the dominant way
  schools, students and teachers are evaluated and
  held accountable.
• Exam preparation is the traditional role of
  assessment in the senior secondary classroom (cf.
  classroom-based assessment).

13
Socio-cultural issues

• Traditional cultural practices exemplified in a
  study of writing assessment in senior secondary
  English (Davison Tang, 2003)
• Many teachers reported they cannot assess,
  only mark. They feel unable to make a
  difference in teaching and learning, to respond
  to individual needs, because of community
  expectations of convergence and commonality.
  Teachers feel their assessment process are
  expected to change, without fundamental purposes
  being explicitly challenged. Such role conflict
  results in increasing stress and a decline in
  perceived teacher expertise.

14
Socio-cultural issues

• However, the official adoption of the UK
  Assessment Reform Groups (1998) distinction
  between assessment for learning, and assessment
  of learning has stimulated the beginnings of a
  major paradigm shift
• Hong Kong schools are moving from a culture of
  testing to a learning and assessment culture
  (Hamp- Lyons 1999 in press).

15
Socio-cultural issues

• Some teachers in the SBA study perceived cultural
  differences as a major stumbling block to
  assessment reform
• I feel it takes time because the culture. The
  education culture in Hong Kong is different from
  other western countries and the students may not
  used to that kind of assessment. They like to do
  exam paper, they think they have something to
  learn.

16
Socio-cultural issues

• Another teacher newly returned from overseas,
  commented
• Actually I was so surprised that ... How slow
  we Hong Kong people are in terms of education
  because I remembered when I was in Canada, we
  never ... You would never be graded on just one
  exam. Its quite like what we are trying to do
  actually, I believe that (assessment for
  learning) has been practised in those places for
  years and I was actually surprised nobody did
  anything (here) I am totally for assessment for
  learning.

17
Socio-cultural issues

• Research shows need for much more understanding,
  even at the system level, of
• the dual role of the teacher as facilitator and
  assessor
• the concept of a student being assessed against
  criteria (rather than other students)
• the need for teacher consultation and interaction
  as part of the standardization process.

18
Socio-cultural issues

• The schools which were already doing extensive
  reading and whose students engaged in oral group
  work and individual presentations on a regular
  basis found it the easiest to integrate the
  assessment tasks into their existing practice
• I just briefly tell the students about the task
  because it is in mid May, so they were quite busy
  that moment. So I asked them just make use of
  what they have been doing, say they just, they
  can just took from ERS report and work on it, say
  prepare a better review so that they can just
  have their presentation based on the review and I
  told them

19
Socio-cultural issues

• Thats what I did at the very beginning. Later
  on, I, I met them some days later, and I asked
  them to show me the book review they had written
  and I took a look at it and I found that there
  werent any major problems in it. So I just
  returned them the review and they started to
  prepare those tasks and later on, just right
  before they did the presentation, I helped them
  with the vocabulary and the names because they
  didnt know how to pronounce them. So I just
  helped them pronounce them correctly (but) I
  gave them more guidance according to the SBA
  documents because the five questions listed
  there (see Appendix II) suggested some sort of
  high order thinking skills So I try to scaffold
  them to think in that way.

20
Socio-cultural issues

• Some teachers took longer to come to grips with
  the principles involved in SBA, and their
  implications for teaching and learning as well as
  for assessment practice
• For students of higher forms, the time (8 min)
  is quite limited. They cant have enough time to
  introduce their books and ask each other
  questions.
• but no time limits actually prescribed in SBA.

21
Socio-cultural issues

• Fairness was and is - a major concern.
• Fairness is fundamentally a sociocultural, rather
  than a technical, issue, a justice that goes
  beyond acting in agreed upon ways and seeks to
  look at justice of the arrangements leading up to
  and resulting from those actions. (Stobart,
  2005, p. 1).
• Fair assessment cannot be considered in
  isolation from both the curriculum and the
  educational opportunities of the students some
  similarities to the more traditional notion of
  fairness embodied in the classical examinations
  for the Chinese civil servicethe concept that
  conditions should be consciously created to make
  opportunities open to all (Hamp-Lyons, 2005).

22
Socio-cultural issues
Fairness in the current HKU examination system Fairness in the SBA component
Treat all students equally (vs. equitably) Treat each student individually (i.e. unequally)
Same task, same input, same conditions, same length of time for all Different arrangements for different students according to their language proficiency
Major aim To give a consistent grade to each student in an invariable settings for selection purposes Major aim To provide opportunity for all students to demonstrate their best. Learning culture for all ? sense of achievement.
23
Socio-cultural issues

• Preliminary findings suggest that these different
  approaches to fairness can be reconciled to some
  extent by providing teachers with opportunity to
  tailor classroom-based assessments to the needs
  of their students, according to commonly-agreed
  processes, outcomes and standards, with teacher
  assumptions about students and their oral
  language levels being made explicit through
  collaborative sharing and discussion with other
  teachers.

24
Theoretical issues

• At the theoretical level concerns with SBA
  revolve around the understanding and
  interpretation of traditional concepts such as
  reliability, validity and authenticity, eg.
• The students will memorize everything.
• Some teachers will tell students the task ahead
  of time
• Teachers will favour their best students.
• Panel chairs will make all teacher agree with
  them.
• Schools will make up results.
• Tutorial schools will coach students.
• The HKEAA should take up all the scripts to
  check the accuracy of the marks.

25
Theoretical issues

• The traditional positivist position on
  language testing, with the tendency to map the
  standard psychometric criteria of reliability and
  validity onto the classroom assessment
  procedures, has been called into question, and
  the scope of validity has been significantly
  broadened (Chapelle 1999 Lynch 2001, 2003
  McNamara 2001) and taken further by a number of
  researchers. (Rea-Dickins, in press)

26
Theoretical issues

• Defining characteristics of school-based
  assessment (Stiggins Conklin 1992 Black
  Wiliam 1998 Brookhart 2003)
• Teacher-mediated
• Co-constructed and dialogic
• Context-dependent
• Multiple and varied
• Dynamic and evolving

27
Theoretical issues

• Such an approach can be seen as constructive
  and enabling because of its focus on assessing
  the process of learning, its attempt to elicit
  elaborated performance, and its emphasis on
  collaborative activity
• Rea-Dickins (in press)

28
Theoretical issues

• However, if there is no reinterpretation of
  traditional conceptualizations of reliability and
  validity, SBA may be reduced to
• a series of summative mini-achievement tests
  external to the teaching and learning programme.
• the assessment of rehearsed monologues or
  dialogues with little or no opportunity for
  authentic language use.
• competition rather than collaboration

29
Theoretical issues

• Cf. Clapham (2000, p. 21)
• Traditional test criteria do apply to
  alternative assessment
• A problem with methods of alternative
  assessment, however, lies with their validity and
  reliability Tasks are often not tried out to see
  whether they produce the desired linguistic
  information marking criteria are not
  investigated to see whether they work and
  raters are often not trained to give consistent
  marks.

30
Theoretical issues

• Preliminary findings suggest
• Potentially much greater validity with SBA than
  external oral paper
• - More natural and authentic tasks
• - Students have a genuine reason to communicate
  as have read different texts
• - Students in comfortable, familiar
  environment-- Class teacher already familiar with
  the range of student performance, can ask
  questions to ensure the text is students own
  work? little possibility of cheating

31
Theoretical issues

• Potentially much greater reliability with SBA
  than external oral paper
• - A series of assessments (instead of one) by a
  teacher who is familiar with the students
• - Exemplars of student performances can
  provide a strong basis for discussing and
  internalizing the set standards.
• - Multiple opportunities for assessor reflection
  and standardization.
• - Disagreement between teachers the key to
  establishing trust (and trustworthiness).

32
Theoretical issues

• But initially lots of misconceptions
• Superficial tasks
• A lot of over-rehearsal and memorization
• Use of buzzers and rigid timelines
• Over-reliance on scripts and formula
• An emphasis on performance and recording
• Overbearing or stressed teachers
• Unhappy students

33
Practical issues

• Not surprisingly, many practical concerns
• - Access to appropriate assessment and extensive
  reading) resources
• - Access to activities and techniques,
  models/resources
• - Access to technical resources/expertise
• - Lack of recognition/support at the
  school-level
• - Concerns about student and parental
  expectations
• - (In)adequacy of training
• - Lack of time
• - Competing priorities

34
Practical issues

• Teachers also expressed a need to understand much
  better the underlying assumptions of SBA, how to
  modify their teaching and learning, and how to
  set up effective assessment tasks
• Many teachers have an urgent need to view
  others practices and share experiences ... We
  can film the good lessons for teachers and
  analyze the lessons. We (need to) focus on
  teaching instead of assessment only.

35
Practical issues

• ? More training will be provided to all teachers
  teaching S4 English between January and April so
  that they can be supported during the whole of
  the assessment period in second semester. SBA
  handbook, introductory DVD and booklet are
  available, on-line and face to face support.

36
Practical issues

• Preliminary findings suggest
• Much extra time and effort required during the
  first year(s) of implementation as SBA is still
  new to both teachers and students, but once SBA a
  routine part of classroom activities, should be
  no significant increase in student/teacher
  workload?
• SBA changes nature of what is done rather than
  how much is done the key is to integrate
  assessment, teaching and learning?

37
Positive impact on teachers, learners and schools

• Despite many concerns, the attitudes towards SBA
  by teachers, students and school communities are
  generally very positive
• Personally, I enjoyed this trialing experience.
  I learnt how to judge the students through this
  activity. Moreover, my students tried to do the
  presentation based on the guiding questions given
  to them. Students found this presentation quite
  interesting and motivating. They learnt how to
  speak confidently and bravely during this
  assessment activity. They found this presentation
  rewarding since they can learn not only from the
  book but also through their actual participating
  experience.

38
Positive impact on teachers, learners and schools

• Those teachers involved now much more aware of
  values and principles underlying SBA, and options
  now available to them
• After my students had finished the
  presentation, I give them feedback and try to
  improve their performance and I was really
  amazed by the response I thought that one of my
  students actually is not very good in English,
  but after the presentation she tried very hard to
  do it again and again. Feedback really works and
  I found that my students have improved a lot.

39
Positive impact on teachers, learners and schools

• Teachers commented on increase in student
  confidence and positive effect of assessment
  activity on other language skills
• M I think my student Sandy, just the one, just
  videotaped. She has great improvement She thinks
  that is useful, very useful and she told me that
  it also helps her reading. I think the student
  can really improve a lot after this trialing.
• C Well, my student has shown a great
  improvement in terms of confidence and English
  proficiency. They like talking to each other in
  English and they are not being afraid of being
  videotaped.

40
Positive impact on teachers, learners and schools

• AYeah, they start, they start reading English
  books as well.
• S I think is just like what you said, is very
  good training for confidence and the students
  actually articulate what they read and they
  found reading very useful, purposeful, that is
  something that you can share with someone, is
  not something just happening in your inner self.
  So I think the one that I trained out of the two,
  one girl and she does quite well even in her
  writing. So she did well in this writing exam.
  Im not sure if it is the effect of that training
  experience (but) it seems that she gained some
  confidence during this period of time.

41
Positive impact on teachers, learners and schools

• Students also commented favorably on the
  assessment activities
• T What do you think about the assessment task
  you did in the presentation task 5?
• S1 It was quite interesting that we need to
  think about what the character needs. We can buy
  a gift.
• S2 I just think its easy to handle it.
• T Why?
• S2 Its quite interesting to think for a gift
  to the character.
• T When you are thinking of a gift is it
  difficult?

42
Positive impact on teachers, learners and schools

• S2 I dont think so because I can think of
  many, many gifts to solve the problem.
• T Did you enjoy working with your partners?
• S2 Yes. I did because my partners are all my
  best friends. We didnt have any gaps so we did
  the project perfect.
• T What about you?
• S3 I also very enjoy doing the task with my
  friends as they know me very much. When I dont
  know what can I say they will help me to continue
  the conversation.
• S4 With the partner I wont feel nervous.

43
Positive impact on teachers, learners and schools

• Students can comment on own development and
  receive constructive feedback immediately after
  assessment, hence improving learning.
• Students encouraged to work consistently.
• Teachers and students become partners in the
  assessment process
• Collaboration and sharing of expertise take place
  within and across schools.
• Teachers build knowledge and skills in
  alternative assessment, readily transferable to
  other areas of English language curriculum, and
  beyond.

44
Conclusions

• The SBA initiative, is a major educational
  reform, entailing significant changes in school
  culture and structures as well as in pedagogic
  expectations among students, teachers,
  administrators, parents and the wider community.
• The SBA initiative requires the development of
  content- and context-appropriate assessment
  activities, instruments and procedures which are
  explicitly linked to high-quality teaching and
  learning.

45
Conclusions

• The SBA initiative requires English language
  teachers who are not only confident and skilled
  at making highly-contextualised, consistent and
  trustworthy assessment decisions, but also
  effective at involving students in the assessment
  process.
• The SBA initiative requires high levels of
  teacher collaboration, leadership and support
  within and across schools.
• SO much research and (informed) practice is
  needed to help make this initiative a success.

46
Assessment for learning in Hong Kong

• http//cd.emb.gov.hk/basic_guide/BEGuideeng0821/ch
  apter05.html
• Based on the beliefs that every student is
  unique and possesses the ability to learn, and
  that we should develop their multiple
  intelligences and potentials there should be a
  change in assessment practices and schools should
  put more emphasis on 'Assessment for Learning' as
  an integral part of the learning, teaching and
  assessment cycle

47

• In other words, teachers should use
  assessments (e.g. as simple as effective verbal
  questioning, observation of student behaviour)
  and provide immediate feedback to enhance student
  learning in everyday classroom lessons. The focus
  is on why they do not learn well and how to help
  them to improve rather than just to use
  assessments to find out what knowledge students
  have learned

48
Selected References

• Adamson, B. Davison, C. Innovation in English
  language teaching in Hong Kong primary schools
  One step forwards, two steps sideways. Prospect,
  18 (1), 27-41.
• Andrews, S. 1994. The washback effect of
  examinations Its impact upon curriculum
  innovation in English language teaching.
  Curriculum Forum, 4 Carless, D., (2005).
  Prospects for the implementation of assessment
  for learning. Assessment in Education, 12(1),
  39-54.
• Chapelle, C. (1999). Validity in language
  assessment. Annual Review of Applied Linguistics,
  19, 254-272.
• Cheng, L.Y. 1998. Does washback influence
  teaching? Implications for Hong Kong. Language
  and Education 11 (1), 38-54.

49
Selected References

• Davison, C. Tang, R. (2003, March). Assessing
  in the swamp Formative assessment in Hong Kong
  secondary school English. Paper presented at the
  American Association for Applied Linguistics,
  Arlington, VA, USA.
• Hamp-Lyons, L. (in press). The impact of
  testing practices on teaching Ideologies and
  alternatives. Classroom-based assessment
  Possibilities and pitfalls. In Cummins, J.
  Davison, C. (Eds) The International Handbook of
  English language teaching, (Vol. 1), Norwell, MA
  Springer.
• Lynch, B. (2003). Language assessment and
  programme evaluation. Edinburgh Edinburgh
  University Press.
• McNamara, T. (2001). Language assessment as
  social practice challenges for research.
  Language Testing, 18(4), 333-349.
• Rea-Dickins, P. (in press). Classroom-based
  assessment Possibilities and pitfalls. In
  Cummins, J. Davison, C. (Eds) The International
  Handbook of English language teaching, (Vol. 1),
  Norwell, MA Springer.
• SBA Consultancy Team (2005). 2007 HKCE English
  examination Introduction to the school based
  assessment component. Hong Kong HKEAA.