Beat the Odds Studies: Connecting Reading and Math Instructional Practice to ValueAdded Results

1
Beat the Odds Studies Connecting Reading and
Math Instructional Practice to Value-Added
Results

• Dave Heistad, Ph.D., Executive Director
• Research, Evaluation and Assessment
• Minneapolis Public Schools
• July 29, 2009

2
Practical Applications of Value-Added in
Minneapolis Public Schools (MPS)

• Study of teachers that beat the odds in 2nd
  grade reading with district-wide self-report
  survey
• Study of Kindergarten teachers that beat the
  odds with video tape of best practice literacy
  instruction in the classroom
• Study of Middle School math teacher teams that
  beat the odds with focus group
• Value-added bonuses for TAP teachers
• Value-added accountability system for school
  rewards

3
I. Value-added Teacher Effects Stability Study
(Heistad, 1999)

• Download district data base of 1993,1994, 1995,
  1996 California Achievement Test (CAT-E) scores
  demographics from mainframe computer
• Zip code poverty levels from census
• Selection of Teachers for analysis
• Taught 2nd grade at least two years in a row
• At least 7 students in a classroom cohort

4
Classroom cohort (students continuously
enrolled)

• Those students taught reading by a single
  homeroom teacher who were enrolled continuously
  throughout the academic school year in question
  and who were tested in reading in the spring of
  that year as well as in the spring of the
  previous school year.

5
Reading Achievement Variable

• Total Reading NCE score from California
  Achievement Test Form E (1993, 1994)
• Total Reading NCE from CAT/5 (1995)
• Reading Comprehension subtest only from CAT/5 -
  Vocabulary was eliminated by the curriculum
  department (1996)

6
Regression equation for Grade 2 CAT-E Total
Reading Value-added
7
Dummy coding

• Value-added teacher effects in the first phase of
  the study were calculated using a dummy code 1
  if the student was in the teachers classroom or
  0 if the student was not in this classroom.

• REGRESSION
• /MISSING LISTWISE
• /STATISTICS COEFF OUTS R ANOVA
• /CRITERIAPIN(.05) POUT(.10)
• /NOORIGIN
• /DEPENDENT read94
• /METHODENTER read93 sex frlunch aframer
  amerind asian hispanic hipovrty bothpar lep sped
  t1 t2 t3

8
(No Transcript)
9
Two-year Teacher Effect Stability
10
Multi-year Generalizability of Teacher Effects
11
Teacher Effect CorrelatesMethodology -
Dimensional

• Approval from MFT
• Teacher assignments verified
• Two self-report surveys administered
• Randomly selected at risk student -22 item on
  specific reading strategies

• Six items on general philosophy of instruction
  and test preparation
• Value-added coefficients merged with 106 teachers
  with complete questionnaires

12
Value-added correlates with general reading
philosophy

• Disagree with the statement, Reading writing
  develop naturally, like speaking
• More small group instruction
• Phonics in grade 2 is critical - both extremes
  correlate

• Report use of some test preparation material
  However no correlation with time spent per week
  no correlation with expensive published materials
  (e.g.. Scoring High)

13
Teachers who beat the odds results

• More development of word attack skills
• More individual student oral reading
• More guidance during initial practice
• More explicit and direct phonics instruction
• More use of systematic motivation strategies

14
One teacher whose students beat the odds

• Ill describe Amandas group...I do direct
  teaching daily (never miss a day)...This class is
  all business and has very tight structure.
• The children love it!!
• Last year my non-readers were reading well by the
  end of the year - most at grade level

15
One teacher whose students beat the odds

• Im going to retire this year, but Ill help -
  because there is such a need.
• Our young teachers dont have any idea how to
  teach reading -- well probably thats an
  overstatement, but children dont learn how to
  read by listening to stories. They learn to read
  by being taught to read.

16
Future research possibilities

• Comparison of statistical models
• Teacher effects for high and low students
• Value-added with other performance measures (e.g.
  K-2 measures)
• Experiment with redistribution of teachers, can
  exceptional teachers turn an at risk program
  around?

17
II. Value-added with Kindergarten Performance
Assessments

• The Minneapolis Public Schools assesses all
  entering kindergarten students in September and
  again in May. The assessment was developed in
  Minneapolis in conjunction with the University of
  Minnesota and is administered by cadre of trained
  retired teachers. Results are reported to
  schools in the fall and used for instructional
  planning. Spring results include the fall
  baseline data to document student growth.

18
Beginning of Kindergarten

• Literacy
• picture vocabulary
• rhyming
• concepts of print
• oral comprehension
• sound recognition
• letter naming
• alliteration

• Numeracy
• counting to 35
• counting backwards
• ordering numbers
• identifying numbers
• before and after
• sums and differences

19
Rhyming

• The rhyming card shows a target picture at the
  top of the card and three additional pictures at
  the bottom of the card. The assessor names all
  of the pictures on the card and the student is
  asked to name or point to the picture at the
  bottom that rhymes with the target picture. It
  is scored as a total number of rhymes correctly
  identified in two minutes.

20
(No Transcript)
21
Alliteration

• The alliteration (beginning sound) card follows
  the same format. The assessor presents the card,
  names the pictures, and the student is asked to
  say or point to the picture at the bottom that
  starts with the same sound as the target picture.
  It is scored as the total number of beginning
  sounds correctly identified in two minutes.

22
(No Transcript)
23
Picture Vocabulary

• Picture vocabulary also uses a set of picture
  cards. These cards have one picture (photographs
  or detailed color line drawings) of an everyday
  object on them. The assessor asks the child to
  name the pictures as fast as they can. It is
  scored as the total number of correct pictures
  named in one minute.

24
Concepts of Print

• A simple book is used to assess concepts of
  print. Students are handed a book and asked to
  show the front of the book, where to begin
  reading, left to right, return sweep and to count
  the number of words on the page.

25
Alphabetic Principle

• Both letter naming and sound recognition are
  assessed using cards that contain randomly
  ordered lower case letters. Both measures have
  been normed in Minneapolis and are fluency
  measures. It is scored as the total number of
  letters named or sounds produced in one minute.

26
Oral Comprehension

• The assessor reads a story to the student and
  asked nine questions. These questions assess the
  childs skill in responding to
• the structure of the story
• inferences
• concepts and vocabulary knowledge
• a personal response.

27
End of Kindergarten

• All of the measures administered in the fall are
  repeated in the spring. In addition passage
  reading is assessed.

28
Passage Reading

• The last item on the end of the year assessment
  is passage reading. Students are asked to read
  pre- primer passage and a total number of words
  read correctly in minute is recorded.

29
Kindergarten Teachers Who Beat the Odds

• were identified empirically using value-added
  analysis
• End of Kindergarten Assessment Results as
  predicted from Beginning of Kindergarten,
  Poverty, ELL, Special Education, Gender, Age, and
  Racial/Ethnic background.
• Ten top teachers were interviewed and video taped
• These teachers worked last summer to produce an
  early literacy instruction video tape

30
Teacher videos are on-line at http//rea.mpls.k12.
mn.us/
31
Middle School Math MCAII Gains and Value-added
Math Growth

• Grade Level Teams that Beat the Odds

32
MCAII Math Growth vs. State Growth Norms Middle
Schools Better than State Growth
33
Value-added

• MCAII Math score greater than predicted for
  grades 5-6, 6-7 and 7-8 using the formula
• 2008 MCAII Math (Scale Score) 2007 MCAII Scale
  Score free/reduced lunch ELL status
  racial/ethnic group Special Education status
  gender math team effect.
• Note Only students enrolled in October and April
  in the same school are included in the analysis

34
Math MCAII 2007 to 2008 Value-added results
Note Anthony had the highest value-added of any
Middle School
35
Beyond zip code analyses What good measurement
has to offer and how it can enhance the
instructional delivery to all students.

• Dave Heistad Rick Spicuzza
• AERA Annual Meeting
• April, 2003 Chicago IL

36
Caution!

• Do not use the Federal Adequate Yearly Progress
  Framework to evaluate the effectiveness of
  Instruction It wont work!

37
Fatal Flaws of a level or means approach.

• Mean test performance is contaminated by factors
  other than school performance.
• Mean performance is out of date, and does not
  address cumulative, compounding effects of
  previous instruction.
• Mean performance is affected by mobility.
• Mean performance interferes with localizing
  performance.

38
Correlation of Poverty and Math NALT Scores
39
Distribution of school average mathematics gain
scores on the NALT by free or reduced price lunch
percentages.
40
A value-added solution

• Value-added models begins to isolate
  statistically the unique contributions schools
  add.
• Account for known empirical influences on
  performance.
• Examine date to identify schools that beat the
  odds.

41
True gain model Growth Curves
42
Improving Student Performance One Student at a
Time
43
NALT, MBST, MCA Different Purposes

• Different tests have different purposes.
  Therefore, different tests are constructed
  differently to meet those purposes.
• MBST makes a pass/fail decision, so most of the
  items are of difficulty level near the decision
  point.
• MCA measures a much wider range of ability but
  has no passing score. Items vary greatly in
  difficulty but all students are administered the
  same set of items.
• NALT/MAP, like MCA, measures a wider range of
  difficulty, but is tailored to the instructional
  level of the student. This allows increased
  precision of measurement while keeping test
  length at a minimum. It also is constructed in
  such a way as to allow for measuring growth
  across time.

44
How is the NALT/MAP Constructed?

• Items are selected from an item bank of over
  60,000 items.
• Most test items were written by teachers. Many
  of the new reading items were written by
  Minneapolis teachers.
• Items for the NALT were selected, from the 60,000
  item-bank, by Minneapolis teachers that best
  reflect Minneapolis curriculum state standards.
• The Measures of Academic Progress (MAP) is the
  computerized version of the NALT which is
  completely adaptive

45
Scoring Example Grade 4 Students Its like the
Olympic diving Competition where degree of
difficulty matters a lot.
46
Grade 4 Growth Example Students who start
furthest behind must make the most gain
47
MPS links for value-added
Quality Performance Awards http//rea.mpls.k12.mn.
us/Quality_Performance_Awards.html School
Information Reports http//www.incschools.com/mpls
/ Supplemental Services Evaluations http//rea.mp
ls.k12.mn.us/Supplemental_Educational_Services.htm
l Kindergarten Teachers Who Beat the
Odds http//rea.mpls.k12.mn.us/BEAT_THE_ODDS_-_Kin
dergarten_Teachers.html