Total Assessment Partnership The Macleans College Perspective

1
Total Assessment PartnershipThe Macleans
College Perspective

• Byron Bentley
• Principal Macleans College

2
Presentation Overview

• School Background
• Background to the TAP Initiative
• Results significant to Macleans College
• Change to Practice
• Our next Step.

3
Macleans College
Macleans College
4
(No Transcript)
5
(No Transcript)
6
Includes China, Hong Kong, Singapore and Taiwan
7

• Macleans Colleges involvement with EAA started
  in 2002.
• We wanted to measure the value we added to our
  students education.
• Most schools struggle to develop a convincing
  method of assessing their contribution to
  students learning, even though research
  generally argues that teachers make a difference.
  

8
Slide of ERO comment

• Students at XXX College receive a sound
  education. Although the overall level of student
  achievement in public examinations is much better
  than the national average, this is to be expected
  given the socio-economic status of the community.
  The Board is not yet able to demonstrate that
  the level of student achievement is as good as
  can be expected given students overall academic
  competence on entry to the school.

9

• Outrageous statement really in that the ERO do
  not have a method for measuring it themselves nor
  give any clear direction.

10

• The results of Macleans College students in
  national examinations have always been high.
• For example in 2005 3 out of the 11 National
  Premier Scholars came from Macleans College.
  (Year 13 award)
• In 2006 2 out of the10 Premier Scholars came from
  Macleans College.
• Our examination success has always been stained
  with the view that as a high decile school, with
  a wealthy parental base, that any achievement, no
  matter how well the students performed, was
  expected.

11

• Our measure of success academically has always
  been national examinations.
• This measures the final outcome, but does not
  measure the improvement of the student during
  their time at school.
• New Zealand has no national examination below
  Year 11.
• EAA satisfies this national measurement need in
  that the examination is sat at Years 9, 10 and
  11.
• EAA is proving to be and will continue to develop
  as a vehicle for the measurement of value added
  achievement.

12
We hold two views of what value added
achievement is at present

• Adding value to the curriculum and developing
  targeted teaching strategies.
• This is leading to refining our assessment and
  reporting practices.

13

• Our goal is to ensure Macleans College is at the
  forefront of education achievement within New
  Zealand and demonstrates the best teaching
  practice on a world wide scale.
• Measuring our rate of improvement in EAA tests
  relative to a nationally consistent cohort is
  problematic because the population is forever
  changing and results are skewed towards high
  ability students who make up the majority who sit
  the tests.
• This needs further work and refinement.
• We firmly believe measuring the rate of
  improvement relative to others is a useful
  measure of school effectiveness.

14
What has Macleans College done?

• Settled on testing our entire cohort at years 9,
  10 and 11 using the EAA competition tests.
• 3 subjects tested. English, Mathematics and
  Science.
• Sequential annual data is important

15
Benefits of EAA Testing

• One of the appealing features of this is that
  there is individual as well as school wide
  benefits.

16
Strength of the Individual
reports

• Parents can have confidence in their childs
  progress
• Highlights strengths and limitations
• This may reinforce known information
• Possibly might be the first nationally comparable
  data they have had
• They are competition tests after all and a large
  number of our students are highly academically
  competitive and thrive on the external challenge
• Useful at parent report evenings, another set of
  data and this hopefully reinforces school
  assessment

17
Macleans Colleges use of the
Individual Report

• Every report is made available to every staff
  member (Intranet)
• Particularly useful for our House Leaders (Deans)
• Every staff member has the capacity to have an
  overview of the performance of every student they
  teach (yr 9 11)

18

• Interesting Data that Macleans College is
  utilizing

19
English - Consecutive Year 9 Cohorts

• Year 9 intake strength has remained consistent
  over time.
• At present this slide contradicts our statements
  that we are a high achieving school however this
  can be explained by the fact that our entire
  cohort enters and this is measured against the
  high achievers from other schools.

20
Mathematics - Consecutive Year 9 Cohorts
2005, 2006, 2007 demonstrates increased ability
on intake in mathematics
21
Science - Consecutive Year 9 Cohorts
22
Summaries

• What follows is a selection of data that we have
  used to change teaching delivery styles or
  emphasis in the curriculum.
• Although our progress comparatively to the
  national cohort shows superficially poor
  progress, the real strength at the moment is our
  ability to compare and contrast selected groups
  within the school.

23

• We are working with EAA to develop a virtual
  population which would mirror the national
  profile of performance if all year 9 10 and 11
  students sat the tests.
• This will allow us to bench mark our performance
  against national standards.

24

• It is interesting to compare results at different
  levels.
• We can then see if the school results change
  compared to the national cohort.

25
Science 2007
Comparison of School Results against the National
Cohort
... but perform less well at higher levels
High ability pupils level at Year 9 ...
26
Science 2007
Comparison of School Results against the National
Cohort
... for the mean as well
The drop at Year 11 needs further examination...
27

• There is a significant change in the national
  cohort at Year 11.

28
Science 2007 - National
Number of students taking the examination
The number of students drops to less than a half
at Year 11
29
English 2007 - National
Number of students taking the examination
The number of students drops to less than a half
at Year 11
30
Mathematics 2007 - National
Number of students taking the examination
The number of students drops to less than a half
at Year 11
31

• It is probable that many of the students that
  drop out at Year 11 are the less able students.
• This means that the Year 11 national cohort is
  further skewed the proportion of high ability
  pupils is greater than at Years 9 and 10.
• This Year 11 factor is seen in many of the
  charts we view.

32
Science 2007 Macleans College
Number of students taking the examination
All Macleans students take the examination at
Year 11
33
Mathematics 2007 Year 11
Macleans students are more than one third of the
national comparison cohort at Year 11
1232
482
34
Strand Analysis
35
Strand Analysis

• Results supplied by EAA are also broken down into
  separate strands for each subject learning area.
• This enables us to see how student performance
  varies in the different strands.

36
A Chart for a Strand
7 questions for this strand
Higher ability pupils are behind the national
cohort
Mean is a bit lower
Lower ability pupils are OK
The strand is English 2007 - Syntax
37
Now to see the progress through the years
38
Progress Year 9 to Year 10
The higher ability pupils have caught up
The mean has dropped back a bit more
Year 10
Year 9
The strand is English 2007 - Syntax
39
Progress to Year 11
.... but some of this can be the Year 11 effect
The higher ability pupils have dropped back again.
The mean gap is higher
... and the lower ability pupils have dropped back
The strand is English 2007 - Syntax
40
Another English strand is Textual Devices
41
Textual devices Years 9 to 11
The lower ability pupils have caught up ..
... And perhaps even moved ahead considering the
Year 11 factor
Year 11
Year 10
Year 9
Behind the national
The strand is English 2007 Textual devices
42
Reading Factual - Years 9 to 11
The higher ability pupils appear to have dropped
back
Year 11
Year 10
Year 9
The strand is English 2007 Reading Factual
43
Reading Literary - Years 9 to 11
The higher ability pupils appear to have dropped
back
We would like to see some improvement here
Year 11
Year 10
Year 9
The strand is English 2007 Reading Literary
44
Vocabulary - Years 9 to 11
We would like to see some improvement here
Year 11
Year 10
Year 9
The strand is English 2007 Vocabulary
45
Similar comparisons can be made with the Science
strands
46
Science Investigating - Years 9 to 11
Good results all the way through
Year 11
Year 10
Year 9
The strand is Science 2007 Investigating
47
Science Observing / Measuring Years 9 to 11
There is a drop-off here
No comparisons can be made as many students
answered all questions correctly
Year 11
Year 10
Year 9
The strand is Science 2007 Observing / Measuring
48
Science Predicting / Concluding - Years 9 to 11
There is a drop-off here
All OK for the higher ability students
Year 11
Year 10
Year 9
The strand is Science 2007 Predicting /
Concluding
49
Reasoning 2005
Weakness in Girls Science reasoning
50
Science Reason / Problem solving - Years 9 to 11
All OK here
Same cohort of students but with improvement
Year 11
Year 10
Year 9
The strand is Science 2007 Reasoning / Problem
solving
51
Science Interpreting - Years 9 to 11
Some problems with Year 11 Interpreting
Year 11
Year 10
Year 9
The strand is Science 2007 Interpreting
52
Mathematics Strands

• Macleans has a high proportion of Asian students
  that are very able mathematicians
• The school also is aware that some of the less
  able students have difficulties with this subject

53
Mathematics Number Arithmetic - Years 9 to 11
No comparisons can be made as many students
answered all questions correctly
Some problems with Number Arithmetic
Year 11
Year 10
Year 9
The strand is Mathematics 2007 Number
Arithmetic
54
Mathematics Space Geometry - Years 9 to 11
No problems with this strand
Year 11
Year 10
Year 9
The strand is Mathematics 2007 Space Geometry
55
Mathematics Measures Units - Years 9 to 11
There appears to be some falling off with lower
ability students
No comparisons can be made as many students
answered all questions correctly
Year 11
Year 10
Year 9
The strand is Mathematics 2007 Measures and
Units
56
Mathematics Chance and Data - Years 9 to 11
There are some problems here with our lower
ability students
No comparisons can be made as many students
answered all questions correctly
Year 11
Year 10
Year 9
The strand is Mathematics 2007 Chance and Data
57
The Common Scale
58
The common scale is a statistical method used by
EAA to give a measure of progress in scores on a
fixed scale.In this way it is hoped to get an
idea of student progress from year to year.
59
The more the shift to the right, the more
progress made
High
Low
This chart is for Science Year 9 through Year 11
60
Scaled Scores for Science
The shift in the school scaled score is less than
the national less progress
High
Low
This chart is for Science Year 9 through Year 11
61
Scaled Scores for Mathematics
Progress 9 to 10 is better than 10 to 11
This chart is for Mathematics Year 9 through Year
11
62
Scaled Scores for English
Notice how there is little improvement between
Year 10 and Year 11
This chart is for English Year 9 through Year 11
63

• For the year 2006 English we have data for five
  years, Year 9 to Year 13

64
English Scaled Scores - 2006
Notice how there is little improvement between
Year 10 and Year 11 and Year 12 to 13
This chart is for English Year 9 through Year 13
in 2006
65

• Choosing groups

66
(No Transcript)
67
The Battle of the Sexes
68
Different groups of students can be compared for
example male and female
69
English Year 11 - Boys
Higher ability boys show some deficiencies
compared to the girls
70
Summary

• The issue with Boys literacy highlighted here is
  consistent with national data and the high media
  profile at present. The issue need addressing at
  Macleans.
• Our Teaching and Learning Team are charged with
  developing strategies to promote literacy across
  the school

71
Mathematics Year 11 Girls
Low ability girls less able in Space and Geometry
and Measures and Units
High ability girls less able in Chance and Data
72
Science Year 11 - Boys
Low ability boys show a small deficiency in
Interpreting skills
... but high ability boys are better
73
It is also possible to define groups for further
analysis
74
Targeted Teaching Strategies

• Drive to improve and up-skill science research
  and problem solving skills
• Very structured projects at years 9 and 10 have
  been introduced. Complete these projects twice
  each year.
• This leads to a more opened ended investigation
  with our Regional Science Fair (Optional) at
  years 9 and 10
• Science club started in 2007 to develop these
  skills in the motivated able students at years
  11, 12 and 13.
• Teachers have access to the question bank to
  examine the nature and style of questions from
  different skills areas. Class sets are
  available.
• Acknowledgement of weakness (relative) and that
  our teaching practice needs to be modified
  accordingly.

75
The Value Added Project Our Continued Development

• We have a wealth of information as to where our
  students stand with respect to each other at any
  one time.
• We also have some indications as to how they
  compare with National groups.

76
In the course of their school career students do
some or all of the following examinations

• School examinations
• The TAP examinations
• PAT tests (Year 9/10 Progressive Achievement
  Test)
• Cambridge CIE examinations
• NCEA assessments

77

• We would like to go one step further and
  determine how our students improve over a period
  of one or more years.
• We are keen to get a fix on Value Added in the
  various subject and strands of our curriculum

78

• We decided to re-visit all of this examination
  data to glean more information about our students
  and our teaching.

79

• A project has been underway over the past year to
  gather historical examination data into a
  specially formulated SQL database.

80
Results Available in the Database
81

• We are able to use this data to chart the
  progress of students over the course of their
  school career.
• One type of chart shows the raw scores for pupils
  in various examinations

82
An Example of a Scores Graph
Isolated scores, taken one year only no line
Pointers for actual CIE results
Science continues as Biology same colour
School examination results
Social Studies continues as Geography same
colour
Legend
83

• Using raw scores has some comparability issues if
  the means in the examinations differ appreciably.
• The database can therefore scale all marks to a
  mean of 60.
• This makes comparisons between in different years
  to be more valid. Simplistic statistical
  adjustment but works well for our use.

84
Comparison of Raw and Scaled Marks
Some smoothing is noticeable
Raw scores
Scores scaled to 60
85

• These charts are very useful in assessing the
  position of students before up-coming external
  examinations.
• They can guide a student as to where extra effort
  would be most useful.
• Pastoral care is also an area where the charts
  are used.
• A third type of chart shows effort scores for
  students over their time at school.

86
An Effort Chart for a Conscientious Student.
1 is good 5 is very unsatisfactory
Good effort scores all the way through nothing
worse than 2.
The lines are staggered so that they do not
overlap.
87
Effort Scores for a Less Serious Pupil
Some problems with effort scores even in Year 9.
Three 4s are not acceptable in Year 13
88

• Yet another chart plots percentiles for the
  student in the subject cohort.

89
Comparison of Raw Score and Percentile Graphs
Shifts become exaggerated
Raw scores
Percentiles
90

• The charts are combined with report comments to
  produce a one page snapshot of a student.
• The snapshots are particularly useful for parent
  interview evening.
• They are also useful in pastoral care situations.

91
An example page
Graph showing marks over the years
Actual CIE scores are indicated
Current teaching sets
Summary of marks
Latest report
92

• We will shortly build EAA results into the same
  system.
• The EAA results for 2003 to 2007 are available

• Raw scores
• Common scale performance
• Percentile
• National percentile
• Strand data

93

• We expect to get very useful information about
  students and cohorts by analysis of the EAA
  results.
• We compare EAA data with results in our school
  examinations and in external examinations.

94
Summary

• EAA results are guiding us to provide targeted
  teaching and resources e.g. science initiative,
  literacy strategy and some targeted mathematics
  teaching to start soon.
• Strand analysis particularly useful in
  highlighting relative weakness and guiding us to
  targeted strategies. EAA provides quite refined
  data.