Fair measures of school performance with a focus on valueadded: challenges and opportunities

1
Fair measures of school performance with a focus
on value-added challenges and opportunities

• Reg Allen
• Tasmanian Qualifications Authority
• Reg.Allen_at_tqa.tas.gov.au

2
the challenges

• statistical models
• not accessible
• unwarranted assumptions

3
multilevel modelling

• the data we have is observational, not
  experimental and not sampled why can we model
  it as if it is the realisation of some random
  variables
• conceptual issues about the meaning of
  inferential outputs have often been bypassed in
  the rush to complex modelling in social science
  research (Draper, 1995) (p.141).

4
a shift in the relationship between research and
politics

• allows the expert to dominate the political
  debate and substitutes expert discourse and point
  of view for public political debate. Indeed,
  politics has become increasingly dominated by
  these experts, who have legitimised ways of
  determining problems in education and their
  solutions. Those in the political arena often
  attempt to use researchers who consider
  themselves experts to legitimise their politics
  and even to explain to those who are recalcitrant
  that they are wrong and that they have no right
  to participate in the debate, because science is
  seen as the only neutral arbitrator, above all
  conflicts of interest (Rochex 2006)

5
effects are causes!

• Raudenbush and Willms, 1995 a school effect is
  the extent to which attending a particular school
  modifies a students outcome
• For consistency with the literature and for
  simplicity in presentation we use the term
  teacher effects when describing the random
  components included in the classroom levels. The
  effects of interest are not necessarily causal
  effects or intrinsic characteristics of teachers.
  Rather, they account for unexplained
  heterogeneity at the classroom level. Ideally,
  they provide information about teacher
  performance but there might be many sources of
  this heterogeneity, including omitted student
  characteristics (Lockwood et al., 2007) (p.127).

6
epidemiology and neighbourhood effects

• spatial variation in morbidity and mortality is
  somehow associated with the clustering of genetic
  predispositions, cultural norms, opportunity
  structures, and/or environmental conditions. By
  definition, advantaged neighbourhoods offer
  cleaner, safer, and less stressful environments
  as compared to, say, ghetto areas. It would be
  shocking to learn that such contexts did not
  somehow impact health. The question is about
  magnitude, mechanism and mutability how big are
  the effects, how do they emerge and how might
  such information be exploited to improve the
  publics health? (Oakes, 2004) (p1929)

7
multilevel modelling

• the impossibility of estimating useful
  neighbourhood effects with a regression model, of
  which the multilevel model is a special case
  (Oakes 2004)
• sometimes adjustment for counfounding doesnt
  work
• the more information put in, the less the effects
  found

8
allow for

• SES?
• race/ethnicity? (England has 18 categories!)

9
modelling carefully done

• can be ok in the general case
• can inform policy by studying general patterns

10
identifying specific effects a school, a teacher

• estimation of treatment effects between groups
  is only reasonable when subjects between groups
  are exchangeable and both groups have some
  non-zero probability of being treated/exposed
  (Oakes, 2006)(p. 644).
• number of parameters and number of cases

11
SUTVA the critical failure

• the assumption that students dont have any
  effect on each other
• simulation study in this what looks like
  teacher effects are actually the result of a
  small number of highly disruptive students who
  affect the learning of other students in their
  class

12
The opportunities

• fair measures of school performance showing how
  value has been added
• the value added has multiple dimensions
• align the measures and the value we want schools
  to add

13
what do we want schools to do?

• Learning in the disciplines will be connected
  with the skills and knowledge required for
  students to prosper and participate in the
  information-rich world of the twenty-first
  century. Students should also participate in
  learning that applies specific discipline-based
  knowledge and skills across disciplines to
  encourage thinking and working in new ways
  (Federalist paper number 2 p. 28).

14
schools add value by

• ensuring that there are increases in students
  disciplined learning
• connecting this learning with the skills and
  knowledge needed for their future economic and
  social prosperity
• increasing the capacity and propensity of
  students to think and work in new ways.

15
an example

• students economic and social futures in the
  twenty-first century require the development of
  awareness, knowledge, and understanding of Asia

16
awareness, knowledge and understanding of Asia

• Australias future economic strength requires
  Australians to be knowledgeable and confident in
  relationships with the peoples of Asia (MCEETYA,
  2006) (p. 6)
• the knowledge, skills and confidence involved
  cant be reduced to a simple set of content
  standards, ready for standardised testing
• developing the required learning is a task for
  schools, experience says its not likely to
  happen if we leave it to others.

17
structural misalignment
18
value-added tables

• can be used by anyone to work out how well the
  school has done
• send the desired signals about what matters.

19
profiles of measures

• compensatory
• clear about anything important left out
• two sets of measures points and letter grades

20
proposed design

• identify the ways in which we expect schools to
  add value
• design measures of all the most important
  features of the ways that a school has added
  value
• develop and publish tables of points that send
  clear signals to schools
• report in terms of a profile of measures
• implement the system progressively.

21
summary

• fair measures of school performance with an
  emphasis of on adding value is a good idea
• real problems with using statistical value-added
  models, deriving individual school or teacher
  effects from complex analyses of limited data
  sets
• statistical limitations
• major structural misalignment
• can do it another way
• clear recognition of and reward for success in
  all the important things