International Comparisons of Mathematics Achievement

1
International Comparisons of Mathematics
Achievement

• Kaye Stacey

2
A growing concern

• Dates approximate different years for data
  collection around the world, publications etc
• FIMSS 1967 IEA
• SIMS 1980 (but maths from an ACER study) IEA
• TIMSS 1994 (third) IEA (Years 4, 8 12)
• TIMSS-R 1999 (repeat)
• TIMSS video study ( 100 lessons in each of 8
  countries)
• PISA 2000 (R m s) OECD (15 year olds)
• TIMSS 2002 (trends) (Years 4 8)
• PISA 2003 (r M s) also problem solving
• TIMSS 2006 (4 yearly cycle being established)
• PISA 2006 (r-m-S) (3 yearly cycle being
  established)

3
Why two series of studies?

• IEA
• all countries on an equal footing, with positive
  and negative consequences (e.g. translation)
• Curriculum based study
• Population defined by school level and class,
  which makes teaching surveys better
• PISA
• Directed to needs of OECD countries ( others)
• Population defined as 15 year olds
• Looking at outcome of schooling - math literacy
  

4
PISA assesses Mathematical Literacy

• Mathematics literacy is the capacity to identify,
  understand and engage in mathematics, and to make
  well-founded judgments about the role that
  mathematics plays in an individual's current and
  future private life, occupational life, social
  life with peers and relatives, and life as a
  constructive, concerned and reflective citizen.
  (OECD PISA)
• A key feature is being able to bring mathematical
  skills to bear in situations which are not
  presented in mathematical terms.
• OECD-PISA divides into 3 classes, from
  reproduction to generalisation, formulation and
  analysis.

5
BOOKSHELVES (FROM PISA 2003, ASSESSMENT FOR 15
YEAR OLDS)

• To complete one set of bookshelves a carpenter
  needs the following components
• 4 long wooden panels,
• 6 short wooden panels,
• 12 small clips,
• 2 large clips and
• 14 screws.
• The carpenter has in stock 26 long wooden panels,
  33 short wooden panels, 200 small clips, 20 large
  clips and 510 screws.
• How many sets of bookshelves can the carpenter
  make?

6
PISA Framework for 2003 Mathematics

• Situations and context
• Everyday and personal
• Workplace and scientific
• Mathematical content - the four overarching ideas
• Space and shape
• Change and relationships
• Quantity
• Uncertainty
• Mathematical processes
• Mathematisation
• The competencies
• Competency clusters
• Reproduction cluster
• Connections cluster
• Reflection cluster
• Assessing mathematical literacy
• Task characteristics
• Assessment structure
• Reporting mathematical proficiency

7
(No Transcript)
8
Australias performance

• Good !
• Plenty of room for improvement !

9
Finnish National Board of Education Adapted from
http//www.oph.fi/english/page.asp?path447,488,42
774

• The strengths of Finnish schools include good
  teachers, efficient teaching and uniform quality.
  The status of the school does not seem to affect
  the level of the students performance.
•  Weakest students raise Finland to the top in
  PISA. The mathematics skills of Finnish students
  are among the best in the world, although
  attitudes to mathematics leave room for
  improvement.
• The skills of Finnish students were among the
  best in all domains assessed in the PISA 2003
  survey, that is, mathematics, reading literacy,
  sciences and problem solving. This outstanding
  achievement is based, in particular, on the good
  performance even by the weakest and average
  students. The average score achieved by the
  weakest quarter of Finnish students was by far
  better than that of the corresponding quarter in
  any other country. The difference to the OECD
  average score in the weakest quarter was 56 in
  mathematics, 61 in science and problem solving
  and as much as 64 in reading literacy. On the
  other hand, Finlands best quarter was the best
  in the OECD only in reading literacy. The best
  Finnish students were surpassed by the best in 4
  countries in mathematics, 1 country in science
  and 3 countries in problem solving. The good
  level of Finlands weakest students is also
  reflected in the smallness of groups at risk.
  With regard to the information society, only 7
  of Finnish students had poor mathematics skills
  (OECD average 21) and 6 poor reading literacy
  (OECD average 19).The Finnish school system has
  succeeded in supporting weaker students. In
  future, more attention should also be paid to
  developing the skills of the better achieving
  children. As a means to this end, PISA
  researchers mention differentiated teaching in
  the teaching group and encouraging talented
  students to develop and use their skills both in
  school and outside it.

10
Finnish National Board of Education Adapted from
http//www.oph.fi/english/page.asp?path447,488,42
774

• The strengths of Finnish schools include good
  teachers, efficient teaching and uniform quality.
  The status of the school does not seem to affect
  the level of the students performance.
• PRINCIPALS AND STUDENTS APPRECIATE TEACHERS
• Finnish principals see their teachers commitment
  and high work ethic as the key strengths of their
  schools. They judge the teachers influence on
  the schools atmosphere to be more positive than
  in the OECD countries on average. In particular,
  principals see their teachers as a positive
  resource in creating a good school environment.
  According to PISA researchers, this is made
  possible because Finnish teachers are quite
  independent and have wide powers of decision
  compared to their colleagues in other countries.
• Finnish students have positive attitudes towards
  their mathematics teachers.
• Mathematics teachers were mainly seen as a strong
  support and guide to the students own learning.
  Mathematics lessons, on the other hand, were
  criticised for being restless. The feeling of
  fellowship in the school community was slightly
  weaker among young Finns than among young people
  in other countries on average.
• FINNISH TEACHING EFFECTIVE
• According to the PISA researchers, Finnish basic
  teaching can be characterised as efficient. The
  time the students spend studying was one of the
  lowest in the countries surveyed. At the same
  time, resources allocated to education are only
  OECD average, so work by students and teachers
  has been very efficient.

11
TIMSS
12
Research questions for TIMSS

• What are mathematics and science students around
  the world expected to learn?
• What opportunities are provided for students to
  learn mathematics and science?
• What mathematics and science concepts,processes
  and attitudes have studentslearned?

13
International Association for theEvaluation of
Educational Achievement

• IEA http//www.iea.nl/
• The International Association for the Evaluation
  of Educational Achievement (IEA) is an
  independent, international cooperative of
  national research institutions and governmental
  research agencies.
• Through its comparative research and assessment
  projects, IEA aims to
• Provide international benchmarks that may assist
  policy-makers in identifying the  comparative
  strength and weaknesses of their educational
  systems
• Provide high-quality data that will increase
  policy-makers understanding of key school- and
  non-school-based factors that influence teaching
  and learning
• Provide high-quality data which will serve as a
  resource for identifying areas of concern and
  action, and for preparing and evaluating
  educational reforms
• Develop and improve educational systems capacity
  to engage in national strategies for educational
  monitoring and improvement
• Contribute to development of the world-wide
  community of researchers in educational
  evaluation
• Since its inception in 1958, the IEA has
  conducted more than 23 research studies of
  cross-national achievement. The regular cycle of
  studies encompasses learning in basic school
  subjects. Examples are the Trends in Mathematics
  and Science Study (TIMSS 1995, TIMSS 1999, TIMSS
  2003, TIMSS 2007) and the Progress in
  International Reading Literacy Studies (PIRLS
  2001, PIRLS 2006). IEA projects also include
  studies of particular interest to IEA members,
  such as the TIMSS-R Video Study of Classroom
  Practices, Civic Education, Information
  Technology in Education (SITES-M1, SITES-M2,
  SITES 2006), and Preprimary Education.
• IEA studies are an important data source for
  those working to enhance students learning at
  the international, national and local levels. By
  reporting on a wide range of topics and subject
  matters, the studies contribute to a deep
  understanding of educational processes within
  individual countries, and across a broad
  international context. In addition, the cycle of
  studies provides countries with an opportunity to
  measure progress in educational achievement in
  mathematics, science and reading comprehension.
  The cycle of studies also enables monitoring of
  changes in the implementation of educational
  policy and identification of new issues relevant
  to reform efforts.

14
IEA current studies

• TIMSS Advanced 2008
• Replication of TIMSS 1995 Advanced Mathematics
  and Physics
• First administered in 16 countries in 1995, TIMSS
  Advanced assesses school-leaving students with
  special preparation in advanced mathematics and
  physics. Countries that...
• TIMSS 2007
• Trends in International Mathematics and Science
  Study 2007
• TIMSS 2007 is the fourth assessment in the
  framework of the Trends in International
  Mathematics and Science Study. The previous
  assessments, conducted in 1995, 1999, and...

15
The largest educational study . . . (1994 data)

• 500 000 students
• 15 000 schools
• 45 countries
• 5 grades in 3 populations
• 1500 items
• 31 languages
• millions of hours and dollars

16
The largest educational study..

• Strengths of the data
• sheer breadth of study
• range of participants
• curriculum, teaching, people, attitudes
• several aspects of achievement
• sampling and quality control measures
• Cautions
• doesnt measure everything
• little depth of information on any one aspect
• needs sophisticated analysis

17
TIMSS 2002 Content Cognitive Dimensions

• Mathematics
• Content Domains
• Number
• Algebra (Patterns and relationships at Grade 4)
• Measurement
• Geometry
• Data
• Cognitive Domains
• Knowing Facts and Procedures
• Using Concepts
• Solving Routine Problems
• Reasoning
• Science
• Content Domains
• Life Science
• Chemistry (combined into physical science at
  grade 4)
• Physics (combined into physical science at grade
  4)
• Earth Science

18
TIMSS 2003 Grade 4 Content Domain Number Main
Topic Fractions and decimals Cognitive Domain
Using Concepts
Click here for TIMSS Grade 4 Mathematics Released
Items
19
TIMSS 2003 Grade 4 Content Domain
Measurement Main Topic Attributes and
units Cognitive Domain Knowing Facts and
Procedures
Click here for TIMSS Grade 4 Mathematics Released
Items
20
Item types

• STUDENTS
• test booklets - rotating and matching
• questionnaires
• multiple choice short answer
• performance tasks
• TEACHERS, PRINCIPALS
• questionnaires

21
SIMS The Second International Mathematics Study
1977- 1981

• The study examined mathematics education at three
  levels  curricular intentions, implemented
  curriculum, and student achievement. Student
  performance was measured and reported separately
  for five areas  arithmetic, algebra, geometry,
  measurement, and statistics. The study included
  some features that involved a replication of the
  First International Mathematics Study (FIMS). It
  also incorporated a detailed longitudinal
  component designed to investigate causal
  relationships between the output and input
  measures of mathematics education.
• Target Population
• This was 13-year-old students and students in the
  final grade of secondary education.
• Participating Countries
• Belgium (Flemish), Belgium (French), Canada
  (British Columbia and Ontario), England and
  Wales, Finland, France, Hong Kong, Hungary,
  Israel, Japan, Luxembourg, Netherlands, New
  Zealand, Nigeria, Scotland, Swaziland, Sweden,
  Thailand, United States.
• Key Findings
• At the middle school level, there was no one
  subtest on which students performed in the same
  general way across the systems. For example,
  students in five systems performed best on the
  arithmetic subtest while those from six other
  systems had their poorest performance on that
  same set of items. One apparent trend was that
  students from any system were more likely to have
  their best or worse performance occurring on
  either arithmetic or geometry than on other
  areas.
• Japan and Hong Kong were the highest achieving
  countries at the final grade of the secondary
  school. Geometry curricula, more than any other
  branch of school mathematics, differed across the
  educational systems.
• At the middle school level, girls tended to
  outperform boys in computational skills and
  algebra. Boys performed better in geometry and
  measurement. By the end of secondary school, boys
  were outperforming girls in every subtest and in
  every country.
• There were many similarities among teachers at
  both grade levels and across systems regarding
  the teaching of mathematics. Teachers were using
  whole-class instructional techniques, relying
  heavily on prescribed textbooks, and rarely
  giving differentiated instruction or assignments.
• Wide differences between systems were observed in
  the degree of opportunity provided to students to
  complete secondary school to the Grade 12 level
  or equivalent. However, during the period between
  the first and the second IEA mathematics studies,
  growth occurred, and in some countries (Belgium,
  Finland) was very substantial (from 13 to 65
  percent and from 14 to 59 percent respectively).
• In the period between the two mathematics
  studies, substantial changes took place in the
  mathematics curricula of many systems. At the
  middle school level, emphasis on arithmetic
  declined while emphasis on algebra and geometry
  increased. At the end of the secondary school
  level, diversity between the nature and extent of
  the mathematics being taught increased,
  particularly with respect to calculus, geometry,
  probability, and statistics.    
• This information is taken from the IEA website
  http//www.iea.nl/sims.html 

22
Trends in Mathematics and Science Study
2003(2001 2004) Extract from IEA website. Key
Findings

• The Asian countries outperformed the other
  participants. Singapore was the top performing
  country at both the fourth and eighth grades in
  mathematics and science. Also Chinese Taipei,
  Hong Kong SAR and Republic of Korea did very
  well.
• At the eight grade countries that showed
  significant improvement in mathematics form 1995
  to 2003 included the Republic of Korea, Hong Kong
  SAR, Latvia (Latvian speaking schools), United
  States and Lithuania. At the forth grade, they
  were Hong Kong SAR, Latvia (Latvian speaking
  schools), England, Cyprus, New Zealand and
  Slovenia. For science, improvement was registered
  in Republic of Korea, Hong Kong SAR, the United
  States, Australia, Slovenia, Lithuania and Latvia
  (Latvian speaking schools) for grade eight,
  Singapore, Hong Kong SAR, England, Hungary,
  Latvia (Latvian speaking schools), New Zealand,
  Slovenia, Cyprus, and Iran.
• Decreases in achievement were found in Japan,
  Belgium (Flemish), the Russian Federation, the
  Slovak Republic, Sweden, Bulgaria, Norway, Cyprus
  (mathematics eighth grade), the Netherlands and
  Norway (mathematics fourth grade), Sweden, the
  Slovak Republic, Belgium (Flemish), Norway,
  Bulgaria, Iran and Cyprus (science eighth grade),
  Japan, Scotland and Norway (science fourth
  grade).
• Gender differences were negligible in many
  countries for mathematics, but in science at the
  eighth grade boys had significantly higher
  achievement than girls in the majority of
  countries. Nevertheless girls had greater
  improvement on average than boys, especially
  since 1999.
• The home context (highly educated parents,
  speaking the language of the test at home, more
  books at home, and frequently using the computer)
  was important to foster higher achievement.
• Providing students the opportunity to learn the
  content assessed was fundamental. The content
  needed to be delivered in the classroom and in an
  effective way.
• Positive school environment was related to higher
  achievement (positive climates for learning,
  fewer students from disadvantaged homes, and
  schools where teachers and students felt safe).

23
Trends in Mathematics and Science Study
2003(2001 2004) Extract from IEA website

• TIMSS was designed to measure trends in students
  mathematics and science achievement. TIMSS 2003
  was the third in a four-year-cycle of
  assessments. TIMSS tests contained questions
  requiring students to select appropriate
  responses or to solve problems and answer
  questions in an open-ended format. From 2003 on,
  TIMSS has gradually placed more emphasis on
  questions and tasks that offer better insight
  into the analytical, problem-solving, and inquiry
  skills and capabilities of students. In addition,
  students, teachers, and school principals in each
  participating country were asked to complete
  questionnaires concerning the context for
  learning mathematics and science, so as to
  provide a resource for interpreting the
  achievement results and to track changes in
  instructional practices.
• The data collection for TIMSS 2003 was conducted
  in OctoberDecember 2002 (Southern Hemisphere)
  and MarchJune 2003 (Northern Hemisphere).
• Target Population
• TIMSS 2003 was assessing the mathematics and
  science achievement of children in two target
  populations. These populations correspond to the
  upper grades of the TIMSS 1995 Population 1 and
  Population 2 target definitions. Generally, these
  are the fourth and eighth grades.
• Participating Countries
• Argentina, Armenia, Australia, Bahrain, Belgium
  (Flemish), Botswana, Bulgaria, Canada (Ontario
  and Québec), Chile, Chinese Taipei, Cyprus,
  Egypt, England, Estonia, Ghana, Hong Kong (SAR),
  Hungary, Indonesia, Iran, Israel, Italy, Japan,
  Jordan, Korea, Latvia, Lebanon, Lithuania,
  Macedonia, Malaysia, Moldova, Morocco,
  Netherlands, New Zealand, Norway, Palestinian
  Autonomy, Philippines, Romania, Russian
  Federation, Saudi Arabia, Scotland, Serbia,
  Singapore, Slovak Republic, Slovenia, South
  Africa, Sweden, Syria, Tunisia, United States,
  Yemen. The Basque Country (Spain) and Indiana
  State (United States) also participated in the
  TIMSS 2003 data collection.

24
Australian Teachers - an area of concern

• Teachers of 13 yr olds
• only 5 under 30
• only 26 think society appreciates their work
• 50 would change to another career if they had
  the opportunity (only NZ higher)
• Teachers of 9 yr olds
• only 35 think society appreciates their work
• 44 would change to another career if they had
  the opportunity (only NZ higher)

25
Results from TIMMS 1994 study

• Mainly from Australian reports on TIMSS published
  by ACER (Lokan, Greenwood Ford, 1995)
• Typical of results from many of the studies, but
  frequently differing in detail.

26
Australian Attitudes to maths
27
Australian Achievement in maths

• 9 year olds above average
• tied for 7th (out of 26) with 6 countries
• a top western, English-speaking country
• 13 year olds above average
• tied for 9th (out of 41) with 14 countries
• a top western, English-speaking country
• Year 12 students did well (in small group of
  countries!)

28
Interesting countries on international ranking

• Singapore, Japan, Hong Kong, Korea
• Netherlands
• 25 years highly professional development of
• Realistic Mathematics Education

29
Features of Australian curriculum

• Seeking wide participation
• Broad curriculum - all main content areas
• Emphasising everyday usefulness including
• embracing calculators
• very informal maths with little proof and rigour
• Aiming to develop strong understanding first,
  then skills
• Advocating active teaching methods

30
Encouraging wide participation

• Language background
• identifying areas of success and of need
• Gender
• no achievement differences for 9 or 13 yr olds
• Home background
• most highly correlated with achievement (0.4)
• Australias results have very high spread

31
Results by content area

• Achievement by content area is generally in line
  with overall performance
• no evidence that a narrow curriculum gives better
  results
• Geometry / Space performance
• extremely high for 9 yr olds
• relatively our worst area for 13 yr olds
• curriculum explanation?
• State differences reflect curriculum emphasis

32
Everyday usefulness in the modern world

• Less emphasis on computation,
• because of advent of calculators
• AND
• Emphasis on strong understanding.

33
Computational skill

• 9 year olds (upper)
• 6000 - 2369 Aust 47 Intl 71
• 23 x 3 Aust at intl average
• 6971 5291 Aust 76 Intl 84
• 13 year olds (upper)
• (8/35) divided by (4/15) Aust 25 lowest
• 6000 - 2369 Aust 85 Intl ?

34
Conceptual understanding of number - 13 yr olds
(upper grade)

• choose largest of 4 fractions
  Aust 43 Intl 39 Sing 76
• order three decimals and a fraction
  Aust 47 Intl 44 Sing
  84

35
Conceptual understanding of number 9 yr olds
(upper grade)

• 0.4 as four tenths
  Aust 48 Intl 39
• 25x18 is how much more than 24x 18
  Aust 40 Intl 45
• smallest whole number made from 4,3,9,1 Aust
  65 Intl 43
• 1/4 pie less than 1/3 pie
  Aust 24 Intl 26

36
0.4 is the same as
37
Which number is the largest?
38
Which number is the smallest?(similar to TIMSS
item, real TIMSS data)
39
Understanding before skills

• Have lower skills, but understanding is not yet
  showing great improvement
• Work on
• activities from which children really learn
• (note Japanese lessons)
• tackle the hard ideas more thoroughly, maybe
  starting earlier

40
Which number represents the shaded part of the
figure? (9 yr olds - upper grade)
41
Differences between the states - Australias most
important data source?

• Significant differences in overall achievement
• Variations in performance by content area (but
  tending to follow overall pattern)

42
Significant differences between states - 13 year
olds (states ordered by achievement)
43
Significant differences between states - 9 year
olds (states ordered by achievement)
44
Why state differences?

• Age and/or amount of schooling ?
• Home background and social factors ?
• Place of maths in total curriculum ?
• Differences in content taught ?
• Teaching practices ?

45
Achievement, age and years at school
46
Teaching practices

• Data difficult to interpret
• Many advocated methods have no straightforward
  relationship to achievement - teacher
  professional development may need to be more
  sophisticated

47
Teaching practices a common pattern

• Primary
• Calculator use
• Having tests
• Secondary
• Using things from everyday life

48
Calculator Use and Achievement
49
Teaching practices - 9 yr olds

• Not working on projects
• Not starting homework in class
• Working on our own from textbooks or worksheets (
  Aust. textbook use very low)

50
Teaching practices - 13 year olds

• Not working on projects
• Not working in groups
• Having homework
• Teacher shows us how

51
Student-reported amount of homework and maths
achievement (13 yr olds)
52
Reviewing Australias report card

• Australia has done well - but
• Is Australia achieving as well as it can?
• Is Australia achieving as well as it needs to?

53
Detailed study of state and international results
to

• Review pace and timing of curriculum
• Design conceptual work better
• Promote aspects of teaching practices which focus
  on student learning
• Review expectation of achievement
• More careful approach to state and national
  curriculum development

54
Kaye StaceyUniversity of Melbourne
55
Embracing technology

• 72 have a computer at home (13 yr olds)
• 80 of principals report computer or software
  shortage
• About 1/3 of 9 yr olds used calculators or
  computers - internationally high