Merging%20Local%20Research%20Data%20with%20PLASC:%20An%20application%20on%20the%20Appropriateness%20of%20Free%20School%20Meals%20as%20a%20Measure%20of%20Deprivation%20in%20Educational%20Research%20in%20Hampshire.

1
Merging Local Research Data with PLASCAn
application on the Appropriateness of Free School
Meals as a Measure of Deprivation in Educational
Research in Hampshire.

• Daphne Kounali and Tony Robinson
• Department of Mathematical Sciences
• University of Bath

2
The HARPS study

• The HARPS project is an acronym for
• Hampshire Research with Primary Schools
• Objective Assess the impact of school
  composition upon student academic progress. The
  main aim of the study is to estimate and better
  understand compositional effects at the primary
  school level.
• Definitions Compositional effects are a subset
  of contextual effects. Compositional effects
  attempt to operationalise the influences of
  fellow pupils characteristics on individual
  pupils performance.
• Compositional variables included in this study
  will be social economic status (SES), ethnicity,
  gender, prior achievement, special educational
  needs (SEN) and age.
• A further remit of the study is to evaluate the
  probity of FSM as a proxy for the economic
  element of SES.

3
Study Design

• The research design is both quantitative and
  qualitative. Like a set of Russian dolls the
  project design is of 3 nested parts
• A large scale analysis of over 300 primary
  schools
• A study of a subsample of 46 schools in the
  Basingstoke and Dean area.
• More detailed case studies of 12 schools.

4
The Basingstoke sub-sample

• One purpose of the Basingstoke subsample is to
  test the sensitivity of the quantitative measures
  and methodologies, including FSM, used to
  identify compositional effects.
• To this end we have collected and are analysing
  detailed family background information from the
  year 3 children in the 46 Basingstoke schools.
  These schools have been selected so as to compare
  the findings from these schools with those of the
  overall Hampshire cohort.
• This will not only test the probity of measures
  such as FSM for the Hampshire dataset but will
  also throw light on their use for school
  effectiveness studies in general.

5

• The Basingstoke subsample dataset contains family
  background data on 1653 pupils from a total of
  2012 students attending 46 out of all 50 schools
  in the Basingstoke and Dean area. Relevant to
  economic status these data include
• Occupational group, Working status, Home
  ownership, Whether in receipt of Working Tax
  Credit, Whether in receipt of FSM.
• Other information was collected on ethnic
  background, family structure, out-of-school
  childcare arrangements and cultural behaviour.
  These data were collected at the beginning of
  2004 for year 3 pupils (7 year old) from these
  schools.

6

• One other measure that is becoming popular in
  research is the Index of Multiple Deprivation
• (IMD)(Noble et al., 2004)
• However this relates not directly to individuals
  but to the small geographical area in which they
  live, known as a low level Super Output Area
  (SOA) containing on average about 1500 people.
  IMD is a composite index based on indices grouped
  within seven domains
• Income Employment Health deprivation and
  disability
• Education, skills and training Barriers to
  housing and services Living environment Crime

7
The Deprivation Geography of Hampshire based on
the Multiple Deprivation Index
Our Basingstoke Schools on the Deprivation Map
The schools are represented by their DFES codes
8

• As a competitor to FSM, the IMD or its income
  component could be used as a distant proxy for
  economic deprivation but technical problems arise
  when it is desired to assign a deprivation index
  to an individual who may or may not share the
  fortunes of an average neighbour.
• In general it is not too difficult to assign a
  deprivation index to a school using IMD
  information from the SOAs from which the school
  draws its pupils. This is then an alternative to
  using the percentage of pupils eligible for FSM
  as an indicator of economic deprivation at the
  school level. Note that the education component
  of the IMD contains achievement data on
  performance at Key Stages 2, 3 and 4 which may
  well preclude it as independent information used
  as input to an analysis of such performance.

9
FSM and how it is used

• FSM is frequently used as a factor in
  representing economic disadvantage in educational
  research
• (studies of attainment or truancy)
• LEAs incorporate FSM figures in calculating SEN
  and AEN provision
• Only statutory available information on economic
  disadvantage
• Used as a proxy indicator of economic
  disadvantage and more generally low SES

10
Current FSM eligibility

• The current eligibility criteria are that parents
  do not have to pay for school lunches if they
  receive any of the following
• Income Support
• Income-based Jobseeker's Allowance
• Support under Part VI of the Immigration and
  Asylum Act 1999
• Child Tax Credit, provided they are not entitled
  to Working Tax Credit and have an annual income
  (as assessed by HM Revenue Customs) that does
  not exceed 14,155
• the Guarantee element of State Pension Credit.
• Children who receive Income Support or
  income-based Job Seeker's Allowance in their own
  right qualify as well.

11
What is FSM

• FSM eligibility is a measure of extreme
  socio-economic disadvantage
• At pupil level indicates the pupils family
  economic disadvantage
• At school level its aggregate indicates
  proportion of families suffering extreme economic
  disadvantage
• It is not a direct indicator of non-economic SES
  disadvantage

12
Criticisms

• The popularity of such uses of FSM eligibility in
  studies in educational research is based on its
  availability. There is no other measure of
  socio-economic disadvantage that is universally
  or even widely available at individual pupil
  level.
• Registration is voluntary affected by shame,
  social class differentials?
• Schools encourage registration because it affects
  school income
• Do families consistently qualify over time?
• Often used as a broad indicator of SES but for
  example does not directly indicate cultural
  capital

13
The use of Large-Scale Data-Bases on Educational
Data

• Opportunities for advantages of using large
  pre-existing data sets
• Conceptual and logistic problems in working with
  these data sets
• Statistical Issues germane to the analysis of
  large, complex data bases

14
The pros

• A means for identifying or enumerating cases
  they can provide the sampling frame can be
  especially helpful in identifying rare events
  i.e. our subsample statistics indicate that 7.5
  incidence of FSM eligibility in the Basingstoke
  and Dean area compared with a yearly estimate of
  9 between 2002-2005.
• Provide a broad, if imprecise overview Frequent
  existing large data bases can be utilized to
  measure and define the magnitude and distribution
  of a problem to more definitive study requiring
  primary data collection.
• This is particularly important in studies in
  social sciences and education where the phenomena
  under study are not themselves directly
  measurable and must be studied indirectly through
  the measurements of other observable phenomena.
• This is true for both the outcomes we are
  measuring
• i.e. we want student progress and we measure test
  performance
• as well as the predictors used
• i.e. we want socioeconomic disadvantage and we
  measure a number of indicators for this including
  FSM.

15

• Could provide directions of the size of
  measurement error in our predictors as well as
  the direction of bias induced by non-response.
  Our Basingstoke subsample was found to have
  higher intakes (baseline tests) in literacy,
  lower incidence of SEN and FSM compared with the
  rest of Hampshire.
• Provide opportunities to Validate the quality of
  research data in terms of what these indicators
  actually measure as well as address missingness.
  In our sample for example non-responders were
  found to be twice more likely to be FSM eligible
  (using the PLASC information)

16
Conceptual and structural pitfalls

• They can be frequently flawed. There are 2
  reasons for this and both related to the fact
  that those who design and compile data bases
  cannot consider the particular needs of every
  person who eventually might use them.
• First, PLASC is primarily used for administrative
  functions related to LEA funding and related
  policies not always explicitly stated and could
  be changing. It is simply not designed or
  maintained to maximize data quality or
  consistency. Codes and records layouts (i.e. SEN)
  may be changed periodically.
• Data elements (i.e. UPNs from the earlier years
  might be unreliable pupil identifiers and the
  same applies for school identifiers after school
  mergings or closings) are often incomplete and
  unreliably coded. Information might be
  inconsistently recorded across LEAs (i.e.
  literacy tests at reception)

17
Pitfalls continue

• Second, even in well-maintained data bases
  (Hampshire LEA with a long history of research
  collaborations) the original goals that dictated
  what data were collected will not coincide with
  those of subsequent users and coding practices
  (i.e. SEN assessed externally) might make
  secondary analysis difficult if not impossible.
  Inclusion of both persons and services might be
  linked to program policies that might change over
  time. Another example is associated with the
  LEAs communication with the individual school
  systems and heterogeneity among schools in the
  type of the actual tests administered (e.g. QCA).
  In most cases we do not know
• How were the data collected and processed
• The time-frame of data collection
• Completeness and coherence among databases of
  different LEAs

18
Another example of complex problems in usage

• Our original population cohort consists of 11793
  (51 boys) of all Hampshire pupils who took the
  baseline test during 2001-2002 and their KS1
  tests during 2003-2004. We have test results for
  approximately 84 of this cohort. The
  Hampshire-wide size of the cohort of Reception
  pupils from PLASC 2001/02 is 14329 and the size
  of the Yr2 pupils from PLASC 2003/04 is 14308.
• The pattern of longitudinal losses in terms of
  test-results seems quite typical for Hampshire
  for this phase, judging for other historical
  cohort data that we also had access to. A wave of
  around 2000 pupils are lost as we move in time
  because they are leaving Hampshire schools and
  another wave of 2000 new pupils are coming-into
  to these schools.
• Because of data-inconsistencies related to
  correctly identifying and recording school
  changes (school merging or closing) we end-up
  looking at 11702 pupils for the phase Baseline to
  KS1 data. These pupils come from 318 (Infant or
  Junior and Primary Schools) at baseline and
  302 schools at KS1.

19
Back to FSM in HARPS and the pros

• Some subsample statistics
• Out of the 1653 Basingstoke families 124 were
  receiving FSM (7.5)
• Of these 13 claimed to receive working tax-credit
• Single parents and non-working couples account
  for the majority of FSM recipients
• So FSM is a very coarse indicator, can we do
  better?
• What about the economically deprived that do not
  fall below the FSM economic threshold?
• No indication of how much the better off
  actually are?

20
A scale of economic status
We estimate a latent score for the unobserved
economic status using relevant data
and Use this Instead of FSM as input to
modelling
X1
Y1
X2
Y2
Latent Index ?
Y3
X3
X1 Working status X2 SES (assessed by
occupation records) X3 education level of the
parent
Y1 Tax-Credit Y2 FSM eligibility Y3 Home
Renting
A graphical representation of the MIMIC model
(Multiple-Indicator-Multiple-Cause)
21
FSM stability

• There is a widely held view that FSM eligibility
  status is stable over time as the following quote
  shows

22
FSM stability in Hampshire 2002-05
23
Who of these ever received FSM?
24
(No Transcript)
25
Observations on FSM stability
26
Assessing the effect of deprivation on school
performance
So, FSM does NOT work
27
More Consequences of the choice of deprivation
indexRanking school Under-Performance

• School Ranks for the Lowest threshold of the KS1
  testing scale
• Comparison of school rankings based on FSM and
  our index suggest that schools can be judged
  performing worse when FSM is used instead of the
  deprivation index.
• So, when we use FSM we tend to under-estimate
  school performance.
• This is because FSM eligibility under-estimates
  the prevalence of economic deprivation.
• Moreover, we also under-estimate the uncertainty
  associated with these rankings.

The differences in rankings affect mostly Average
(typical) performing schools
The plots depict the school-level residuals
(probit scale) against their rank-order. Large
negative values in the probit scale characterize
the best performing schools.
28
The area where FSM under-estimates school
performance is below the diagonal line.Higher
Residual Ranks indicate Worse Performance

• 61 - 71 (depending on the residuals considered
  for each threshold of the KS1 scale)
• of the schools are judged as performing worse
  when using FSM rather than using our index.
• Moreover, the bias also seem to be differential
• More deprived schools (in terms of intake and/or
  FSM eligibility) being affected more.

29
Derivation of the index of disadvantage

• A MIMIC model (Multiple-Indicator-Multiple-Cause)
  model was used for the derivation of a scale
  representing economic disadvantage. Binary data
  on three indicators of economic disadvantage
• Y1 working tax-credit, Y2 FSM eligibility, Y3
  Home renting) are used
• in a two-parameter logistic model (Rasch-model)
  as follows
• logitPr(yij??j) dij?i dij ?i?j, for indicator
  i1,2,3 and subject j
• where di is a three-dimensional vector
• for the i-th element equal to 1 and all other
  elements equal to zero
• and ? is the latent unobserved measure of
  economic disadvantage.
• The parameter estimates for ?i (location
  parameters, or intercepts) represent the potency
  of each indicator.
• The parameters ?i (factor loadings) represent the
  discriminating ability of each indicator.

30

• We also specified a structural model for the
  latent economic disadvantage, allowing the mean
  indicator potencies to differ between groups as
  follows
• ?j?0?1X1?2X2?3X3?di1X1,
• where X1 working status, X2 SES assessed from
  occupation records, X3 level of education
• The term ?di1X1 allows us to introduce a direct
  effect of working tax credit on the receipt of
  working tax credit, apart from its indirect
  effect through the latent variable.
• This is needed since only employed people would
  be
• in receipt of tax-credit.

31

• The intercepts estimates suggest that the
  estimated indicator potencies increase from FSM
  to renting with FSM being the most difficult
  benefit to get. FSM eligibility does have the
  highest discriminating ability (slope) as
  expected followed by home renting.
• As expected the estimated parameter ? indicates
  that the odds of receipt of of working tax-credit
  are significantly reduced for non-working couples
  compared with the typical couple where one is
  working full-time and the other part-time.

32

• Higher SES reduces economic disadvantage compared
  to middle classes
• Low or unknown SES increases economic
  disadvantage.
• Higher education level is associated with reduced
  economic disadvantage.
• Non-working couple are significantly more likely
  to be economically disadvantaged, as expected,
  compared with the typical working status of a
  couple in this sample (one working full-time and
  one part time) with the single part-time parents
  following closely.
• Couples consisting of partners where only one
  full-time working time are also significantly
  more disadvantaged compared to the typical
  working couple which appears similar to couples
  where both partners work full-time.
• It is interesting to note that couples consisting
  of partners where one is working full-time and
  the other is self-employed, are significantly
  less disadvantaged compared with the typical
  couple

33

• Working tax-credit is a much
• More useful indicator especially
• For middle class occupations

• FSM refers to
• very disadvantaged children
• even among low SES families

34

• Receipt of Working tax credit
• as an indicator of economic disadvantage seem to
  INDEX
• occupation based SES, MOSTLY but not always.
• This might be due to bias associated with
• reporting occupation and/or
• assumptions on occupation rankings.

• Receipt of Working tax credit
• as an indicator of economic disadvantage seem to
  INDEX
• level of parental education less well.
• It seems that level of education is consistent
  with economic disadvantage for the very poor.
• The economic value of vocational qualifications
  seem less clear for more typical incomes.
• This could also be due to the fact that in our
  sample
• 90 of the responders are women and only 60 of
  these women contribute to the determination of
  SES.

35
Further developments

• Improvements in the construction of the index
• Allow for differential measurement error through
  joint modelling instead of plugging-in index
  estimates.
• In the structural part of the model include
  interaction terms for education level and the
  identity of the major bread-winner.

• Address problems related to endogeneity i.e.
  economic disadvantage
• might affect both school intakes as well as KS1
  test results,
• and there might be both direct and indirect
  effects
• of economic disadvantage on school performance.

36
Conclusions
37

• As far as the use of PLASC data is concerned the
  conclusion is
• Documentation and Documentation and
• The importance for the research community to get
  involved with the development and progress of
  maintaining and improving such invaluable data
  resources as the PLASC data.