FLORIDA COUNCIL of LANGUAGE ARTS SUPERVISORS

1

FLORIDA COUNCIL of LANGUAGE ARTS SUPERVISORS
NAEP Reading October 2006 Michele Sonnenfeld NAEP State Coordinator Florida Department of Education
2
National Assessment of Educational Progress
(NAEP)

• The Nations Report Card
• NAEPs two major goals are to
• Measure and compare student achievement in states
  and other jurisdictions
• 2. Report changes in achievement of 4th, 8th, and
  12th graders over time

3
NAEP

• Has produced National results for Grades 4, 8,
  and 12 since 1969
• Has produced state-level results at Grades 4 and
  8 since 1990
• Has produced results for the Trial Urban District
  Assessment (TUDA) since 2002
• Does not produce student-, school-, or
  district-level results, except for the 10 TUDAs

4
Current OrganizationalStructure of NAEP
5
Federal Legislative Reference to NAEP

• No Child Left Behind Act (NCLB) of 2001
• Section 111(c)(2) of NCLB requires
  states/districts who receive Title I funding to
  participate in biennial State NAEP assessments of
  4th and 8th graders in Reading and Mathematics,
  beginning with the 2002-2003 school year.

6
State Legislative Reference to NAEP

• Florida State Statute 1008.22 (2)
• The Commissioner of Education shall direct
  Florida school districts to participate in the
  administration of NAEP . . . both for the
  national sample and for any state-by-state
  comparison programs .
• State legislation requires all selected schools
  to participate in all NAEP assessments.

7
NAEP Components
STATE (Public) (Grades 4 8)
8
NAEP Schedule of Assessments
NAEP Schedule of Assessments NAEP Schedule of Assessments NAEP Schedule of Assessments
Year National State
1996   Mathematics (4,8) Mathematics (4,8)
1996   Science (8) Science (8)
1998     Reading (4,8) Reading (4,8)
1998     Writing (8) Writing (8)
1998     Civics (4,8,12)  
2000     Mathematics (4,8) Mathematics (4,8)
2000     Science (4,8) Science (4,8)
2000     Reading (4)  
2002   Reading (4,8) Reading (4,8)
2002   Writing (4,8) Writing (4,8)
2003   Reading (4,8) Reading (4,8)
2003   Mathematics (4,8) Mathematics (4,8)
2005     Reading (4,8,12) Reading (4,8)
2005     Mathematics (4,8,12) Mathematics (4,8)
2005     Science (4,8,12) Science (4,8)
2006     U.S. History (4,8)  
2006     Civics (4,8)  
2006     Economics (12)  
2007     Reading (4,8,12) Reading (4,8)
2007     Mathematics (4,8,12) Mathematics (4,8)
2007     Writing (8,12) Writing (8)
2009     Reading (4,8,12) Reading (4,8)
2009     Mathematics (4,8,12) Mathematics (4,8)
2009     Science (4,8,12) Science (4,8)
9
NAEP Frameworks

• The American Institute of Research (AIR), with
  the assistance of teachers, develops the NAEP
  frameworks.
• The National Assessment Governing Board (NAGB)
  approves and adopts the frameworks after they
  have been reviewed and critiqued by teachers.
• Each framework describes the content and format
  of the 4th, 8th, and 12th grade assessments.
• There are frameworks for each of the content
  areas assessed by NAEP.

10
NAEP Reading Framework

• Three Contexts for Reading
• Reading for literary experience
• Reading for information
• Reading to perform a task
• Four Aspects of Reading
• Forming a general understanding
• Developing interpretation
• Making reader/text connections
• Examining content and structure

11
NAEP Design Item Development

• Items are based on the subject-area frameworks
  approved by the National Assessment Governing
  Board (NAGB).
• Items are developed by teachers, subject- area
  specialists, and assessment experts.
• Items meet professional standards for testing.
• Experts review items for technical/statistical
  quality, content quality, bias, and sensitivity.

12
How is Floridas NAEP School Sample Selected?

• Step 1 Lists of schools are obtained from the
  National Center for Education Statistics Common
  Core of Data (CCD).
• Step 2 Schools are grouped into strata by two
  primary characteristics
• Type of location (central city, urban fringe,
  rural)
• Minority enrollment (white, African American,
  Hispanic)
• Step 3 Within strata, schools are ordered by a
  measure related to achievement.
• Step 4 A proportional sample is selected
  systematically from the ordered list.

13
Measure of Achievement Used for Floridas NAEP
School Sample Selection

• Grade 4
• Mathematics average scale scores for all Grade 4
  students in 2005
• Grade 8
• Median household income (because the match rate
  for achievement scores for 8th grade was less
  than 70)

14
How is Floridas NAEP Student Sample Selected?

• A list of all 4th or 8th grade students enrolled
  at the sampled schools is submitted
  electronically to Westat from the FLDOE, based on
  October FTE data. Westat randomly selects
  students chosen to participate in NAEP.
• 90 of selected students at each chosen school
  must participate in the assessment. Make-up
  sessions are scheduled at the schools not meeting
  this benchmark.
• Students exempted by parental request are not
  replaced and are not used to calculate the
  assessment participation rate.

15
NAEP 2007

• NAEP 2007 administered in Florida from January
  22 through February 21, 2007.
• Each student takes only a portion of the
  questions, answering two 25-minute sections of
  subject-area questions.
• Students also answer questions related to their
  backgrounds and their home or school experiences
  in the subject of their assessment.
• Assessment is administered by a team from Westat,
  the NAEP administration contractor for the US
  DOE, and the National Center for Education
  Statistics (NCES).
• Parents must be notified prior to the assessment.
  Refusals must be in writing.

16
NAEP Sampling School and Student Selection

• 350 Florida public schools in 49 districts will
  participate in NAEP in 2007.
• Grade 4 Reading and Mathematics
• Floridas sample will include 167 schools.
• Schools can elect to take all students (up to
  120).
• If there are more than 120 4th graders, NAEP
  will select a sample of 62 students to assess.
• Grade 8 Reading, Mathematics, and Writing
• NAEP will select a sample of 93 students at each
  of the 161 Florida schools included in the
  sample.
• Grade 12 Reading and Writing
• Floridas sample will include 31 high schools
  and
• 75 students from each school will be selected to
  participate.

17
NAEP Inclusions and Accommodations

• NAEP has included students needing accommodations
  since 1998.
• Students with disabilities (SD) and English
  language learners (ELLs) are to be included in
  NAEP based on an IEP/504 Accommodations Plan/LEP
  Committee decision.
• Most frequent accommodations with NAEP
• Extra testing time
• Individual or small group administrations
• Large print booklets
• Does not include reading aloud of passages or
  questions on the NAEP reading assessment.
• Assessment must be completed in one day.

18
NAEP Analysis and Reporting

• National, State, and 10 Trial Urban District
  Assessment (TUDA) results.
• NAEP scores are reported in two ways by Scale
  Scores and by Achievement Levels
• No student, classroom, or school results are
  provided
• No district results are provided except for the
  10 TUDAs
• Subgroup, state, and national comparisons can be
  made
• Relationships between student proficiency and
  certain background variables are available

19
NAEP Scale Scores

• Scale scores provide a comparison of student
  performance with regard to a group (e.g., the
  nation) or a subgroup (e.g., ELL) on a set of
  items.
• Scale scores
• Reading and Mathematics scale score range is
  0-500
• Science and Writing scale score range
• is 0-300
• Scale scores are reported as an average for
  groups of students and may obscure progress or
  problems related to student performance across a
  distribution of scores.

20
National NAEP Average Scale Scores for Grades 4
and 8 Reading 1992-2005

• National NAEP reading average scale scores are 2
  points higher in 2005 than in 1992 at both Grades
  4 and 8.

21
Florida Compared to the Other 49 States, District
of Columbia, and Department of Defense
SchoolsNAEP 2005 Average Scale Scores Grade 4
Reading
Of the 50 states and two jurisdictions that
participated in the NAEP 2005 Grade 4 reading
assessment, students average scale scores in
Florida were higher than that of 15, not
significantly different from that of 19, and
lower than that of 17.
22
Florida Compared to the NationNAEP Average Scale
Scores 1998-2005 Grade 4 Reading
Florida improved from scoring below the National
average in 1998 to above the National average in
2005 on NAEP Grade 4 reading.
23
Florida Compared to the Other 49 States, District
of Columbia, and Department of Defense
SchoolsNAEP 2005 Average Scale Scores Grade 8
Reading
Of the 50 states and two jurisdictions that
participated in the NAEP 2005 Grade 8 reading
assessment, students average scale scores in
Florida were higher than that of 5, not
significantly different from that of 11, and
lower than that of 35.
24
Florida Compared to the NationNAEP Average Scale
Scores 1998-2005 Grade 8 Reading
Florida NAEP Grade 8 reading performance
parallels and continues to score below the
National performance.
25
NAEP Grade 4 Reading 1998-2005Florida and the
Nation Average Scale ScoresRace/Ethnicity
26
NAEP - Florida and the Nation Grade 4 Reading
1998-2005

• Performance Gaps Between Average Scale Scores
• by Race/Ethnicity

27
NAEP Grade 8 Reading 1998-2005Florida and the
Nation Average Scale ScoresRace/Ethnicity
28
NAEP - Florida and the Nation Grade 8 Reading
1998-2005

• Performance Gaps Between Average Scale Scores
• by Race/Ethnicity

29
NAEP Achievement Levels

• NAEP Achievement Levels (ALs) try to identify
  what students should know and be able to do at
  various achievement levels.
• Advanced, Proficient, Basic, and below Basic
  indicate NAEP achievement levels. Each is
  determined by cutpoints established by
  professional educators and NAGB.
• The number or percentage of students in the
  population at large or in subpopulations can be
  reported by the four achievement levels.
• Results show how different groups are performing
  in relation to each other and over time.

30
Florida Compared to the Other 49 States, District
of Columbia, and Department of Defense
SchoolsNAEP 2005 at Basic and above Grade 4
Reading
Of the 50 states and two jurisdictions that
participated in the NAEP 2005 Grade 4 reading
assessment, the percentage of students scoring at
Basic and above in Florida was higher than that
of 15, not significantly different from that of
19, and lower than that of 17.
31
NAEP Achievement Levels Florida and the
NationGrade 4 Reading 1992-2005
Florida has a greater percentage of students
scoring at Basic and above in 2005 than does the
Nation on NAEP Grade 4 reading.
32
Florida Compared to the Nationat Basic and above
Grade 4 NAEP Reading 1998-2005
Florida went from scoring below the National
average in 1998 to scoring above the National
average in 2005 at the Basic and above
achievement level on NAEP Grade 4 reading.
33
Florida NAEP Grade 4 Reading Summary of
Achievement Level Results

• The percentage of students in Florida who
  performed at or above the NAEP Basic level was 65
  percent in 2005. This percentage was not
  significantly different from that in 2003 (63
  percent) but greater than that in 1992 (53
  percent).
• Florida is one of six states that had more than a
  10 point increase in the percentage of low-income
  students who performed at the Basic level or
  higher between 1998 and 2005.

34
Florida Compared to the Other 49 States, District
of Columbia, and Department of Defense
SchoolsNAEP 2005 at Basic and above Grade 8
Reading
Of the 50 states and two jurisdictions that
participated in the NAEP 2005 Grade 8 reading
assessment, the percentage of students scoring at
Basic and above in Florida was higher than that
of 4, not significantly different from that of
13, and lower than that of 34.
35
NAEP Achievement Levels Florida and the
NationGrade 8 Reading 1998-2005
Florida has a smaller percentage of students
scoring at Basic and above than does the Nation
on NAEP Grade 8 reading.
36
Florida Compared to the Nationat Basic and above
Grade 8 NAEP Reading 1998-2005
Florida NAEP Grade 8 reading performance
parallels and continues to score below the
National performance.
37
Florida NAEP Grade 8 Reading Summary of
Achievement Level Results

• The percentage of students in Florida who
  performed at Basic and above was 66 percent in
  2005. This percentage was not significantly
  different from that in 2003 (68 percent) and in
  1998 (67 percent).

38
FCAT and NAEP

• FCAT measures student performance on selected
  benchmarks, as defined by Floridas Sunshine
  State Standards (SSS)
• NAEP reports on the performance of groups of
  students at a given time - and across time -
  without specifying how a subject should be taught

39
FCAT and NAEP ParticipationGrade 4 Reading 2005
All Students White Students African American Students Hispanic Students
FCAT 195,680 93,746 44,857 45,840
FL NAEP 4,247 2,039 977 977
National NAEP 156,766 89,357 26,650 29,786
40
FCAT and NAEP ParticipationGrade 8 Reading 2005
All Students White Students African American Students Hispanic Students
FCAT 201,760 100,920 46,349 45,579
FL NAEP 3,821 1,949 879 802
National NAEP 150,568 90,341 25,597 25,597
41
FCAT, Florida NAEP, and National NAEPGrade 4
Reading 1998-2005
Achievement Levels FCAT Level 3 and above
and NAEP Percent Basic and above
42
FCAT, Florida NAEP, and National NAEPGrade 8
Reading 1998-2005
Achievement Levels FCAT Level 3 and above
and NAEP Percent Basic and above
43
FCAT and Florida NAEP Grade 4 Reading by
Achievement Level 1998-2005
Percentage of students scoring at Level 3 and
above on the Grade 4 reading FCAT and Basic and
above on Florida NAEP increased between 1998 and
2005. There is a 24-point gain on the FCAT and
12-point gain on Florida NAEP.
44
FCAT and Florida NAEP Grade 8 Reading by
Achievement Level 1998-2005
Performance of Grade 8 students in reading has
been flat on both the FCAT and Florida NAEP.
There is a 5-point gain on the FCAT and 1-point
loss on Florida NAEP between 1998 and 2005.
45
FCAT and Florida NAEP by Achievement LevelGrade
4 Reading 1998-2005 Race/Ethnicity
White, African American, and Hispanic students
all improved on Grade 4 reading from scoring
lower on the FCAT than on Florida NAEP in 1998
to higher on the FCAT than on Florida NAEP in
2005.
46
FCAT and Florida NAEP by Achievement LevelGrade
8 Reading 1998-2005 Race/Ethnicity
White, African American, and Hispanic students
continued to score higher on the Grade 8 reading
Florida NAEP than on the FCAT.
47
The Nations Report Cardhttp//www.nces.ed.gov/na
tionsreportcard/
48
NAEP Websites

• Floridas NAEP website
• http//firn.edu/doe/sas/naephome.htm
• NAEP website
• http//nces.ed.gov/nationsreportcard/
• NAEP Data Explorer
• http//nces.ed.gov/nationsreportcard/nde/
• NAEP Questions Tool
• http//nces.ed.gov/nationsreportcard/itmrls

49
NAEP Coordinator Contact Information

• Michele Sonnenfeld
• NAEP State Coordinator
• Florida Department of Education
• [phone]
• SC 205-0787
• FAX [phone]
• SC 205-0781
• Michele.Sonnenfeld_at_fldoe.org