National Assessment of Educational Progress NAEP and the NAEP Toolkit http:nces'ed'govnationsreportc

1
National Assessment of Educational Progress
(NAEP) and the NAEP Toolkit?lthttp//nces.ed.gov/n
ationsreportcardgt
Sandy Garrett State Superintendent of Public
Instruction Oklahoma State Department of Education
Cathy Douglas, Language Arts Director Penny
Gooch, Reading and Literacy Director Melissa
Jones, NAEP State Coordinator Kerri White,
Mathematics Director
2
Videoconference Agenda

• Overview
• What is NAEP?
• NAEP Resources
• NAEP Reading Resources
• NAEP Writing Resources
• NAEP Math Resources
• NAEP Science Resources
• Questions

3
Introducing NAEP The Nations Report Card

• The Nations only monitor of what students know
  and can do in various subject areas
• A survey designed to produce national and state
  level results it is not a testing program for
  individual students or schools
• Produces results for populations, not for
  individuals
• Student and school results are confidential

4
NAEP Components
The National Assessment of Educational Progress
MAIN
LONG-TERM
National Public Private Grades 4, 8 12
National Public Private 9, 13 17 yr olds
State (Public) (Grades 4 8)
District -Trial (Public) (Grades 4 8)
5
The Nations Report Card 2003 Assessment
Estimates

• In a six week period . . .
• Nearly 15,000 schools
• Nearly 850,000 students
• About 500 schools a day

6
NAEP Design Features

ASSESSMENT STAGES
Content Frameworks and Test Specification
Item Development
Analysis and Reporting
Sampling and Data Collection
Pilot/Field Testing
7
Schedule of Assessments
Approved by NAGB Aug. 2, 2003
8
NAEP

• No Child Left Behind legislation stipulates that
  states, districts, and schools within districts
  that receive Title I funds must participate in
  NAEP if selected.
• Reading and Mathematics must be administered
  every two years in Grades 4 and 8.
• In 1998, Oklahoma lawmakers passed a law
  requiring our state to participate.

9
State Law

• Title 70 Oklahoma Statutes 1210.513
• To ensure the participation of the Oklahoma
  public school system in the National Assessment
  of Educational Progress. The results of this
  assessment shall be included as a separate result
  in annual reports on the Oklahoma State Testing
  Program.

10
NAEP Design Content Frameworks

• Frameworks and test specifications are the
  responsibility of NAGB
• Process provides an organizing framework for the
  content of each subject
• Both are developed by a national consensus
  process
• Frameworks and specifications guide the
  development of the test questions

11
NAEP Design Item Development

• Items based on the subject area frameworks
• Developers include teachers, subject area
  specialists, and testing experts
• Meets professional standards for testing
• Experts review items for quality, bias, and
  sensitivity

12
NAEP Design Sampling and Data Collection

• Multi-Stage Stratified Random Sampling
• Selection of schools List obtained from the
  NCES Common Core of Data (CCD)
• Selection of students within schools
• Sample Size for State NAEP years
• Approximately equal sample sizes for each state
• About 100 schools for each grade and subject
• About 3,000 assessed students for each grade and
  subject

13
Sampling and Data Collection (continued)

• State level results are reported for Grades 4 and
  8. Grade 12 results are reported at the national
  level.
• Assessments will take approximately 90 minutes of
  student time.
• Students will only be asked to take one content
  area.
• Math, Reading, and Writing can be given in the
  same setting.

14
Sampling and Data Collection (continued)

• Additional Samples
• Grade 12
• Long-term trend
• Field trials
• Special studies
• Sample sizes vary in proportion to the student
  population of the state
• Contractor Administration

15
Instrument Design

• Long Test, Short Booklet
• Each student gets a small part of the test
• No individual student scores
• Matrix sampling
• Test Questions
• Multiple-choice
• Open-ended
• Background Items

16
NAEP Compared to Oklahoma CRTs

• NAEP
• Timed Test
• Multiple Choice, Short Answer, Essay
• Student Sample
• Based on NAEP Framework
• Cumulative
• Not Used for Accountability
• State- and National-Level Results

• OKLAHOMA CRTs
• Untimed Test
• Multiple Choice (except for Writing)
• All Students
• Based on State Standards
• Grade-Specific
• Used for Accountability
• Student- and School-Level Results

17
NAEP Design Analysis and Reporting

• Ways of Reporting Results
• Scale Scores
• Achievement Levels
• Data Disaggregated by
• Subgroups (e.g., race, gender, school lunch, SD,
  LEP)
• State and regional comparisons
• Background factors (i.e., school, teacher, and
  student factors)

18
Analysis and Reporting (continued)

• Scale Score
• 0 - 500
• Achievement Levels
• Advanced
• Proficient
• Basic
• Below Basic

19
Analysis and Reporting (continued)

• Reporting Formats
• Nations Report Cards and Highlights
• Snapshot reports for states and districts
• Reports on Special Topics
• Web Releases
• summary data tables / NAEP Data Tool
• released items and responses
• electronic publications
• State Report Generator

20
Using NAEP

• Who uses NAEP?
• Educators
• Policymakers
• Media
• And for what purposes?
• Tracking trends
• Comparisons across states
• Subgroup comparisons in states
• and the nation

21
Why the participation of our students is
important!

• Each selected student represents hundreds of
  students across the nation!
• Important decisions about education are made
  using the results of these tests.
• Students have the ability to contribute to
  improving the quality of education for every
  student in our state and the country.

22
Resources Available

• NAEP Questions Tool
• The Questions Tool provides easy access to NAEP
  questions, student responses, and scoring guides
  that are released to the public. Both national
  and state data, where appropriate, are presented.

23
NAEP Questions Tool

• Provides educators, researchers, parents, and
  students with access to released questions from
  NAEP assessments in nearly all assessments,
  dating from 1990.
• Provides examples of what NAEP asks students in
  Grades 4, 8, and 12.
• Examines student performance on a specific
  question by subgroups and state.
• The NAEP Questions Tool is available online at
• lthttp//nces.ed.gov/nationsreportcard/itmrlsgt.

24
(No Transcript)
25
Questions by Subject
26
Sample Questions
27
NAEP Questions Tool
28
NAEP Questions Tool
29
NAEP Questions Tool

• Provides list of questions available
• Select question desired or next group of
  questions
• Can order questions by all of the variables
• Click box in the In Folder column for questions
  you want to add to Print Folder

30
NAEP Questions Tool
Click the question description to select the
question
Click box to add question to print folder
31
NAEP Questions Tool

• Actual question
• Performance data (national)
• Content classification
• Scoring Guide/Key
• Student responses at each score level (scorers
  commentary)
• Additional data (subgroup performance, state
  performance, etc.)

32
Click the Question tab to view the actual
question
View the reading passage by clicking here
Click here for a printable version of the question
33
The reading passage
34
The second tab provides National performance
results for this question
35
The content classification tab provides
information about how the question relates to the
subject area framework. Includes descriptions of
the content domain (whats being assessed) and
the cognitive skills within that domain.
36
Describes range of scores and criteria used to
judge response for questions that ask student to
write a response
37
Displays sample student responses at each score
level for questions that ask students to write a
response
38
Provides explanation of the score given
39
Shows additional data for each question by
various subgroups (gender, race/ethnicity, Title
I, etc.)
Also, provides results for each state
40
(No Transcript)
41
Questions Tool Print Folder Option

• Allows you to. . .
• Add,
• Customize, and
• Print questions that you select.

42
Adding Questions to Print Folder

• From individual question screen, click Add
  Question
• From Search Results screen, click the box in
  the In Folder column for each question you want
  to print or click Add All Questions to add all
  questions from list
• Click To Print Folder to view contents and use
  options

43
NAEP Questions Tool
44
Remember to Save after editing
45
Customizing Print Options

• Select options you want from Print Options
  check boxes (questions, performance data, scoring
  guides/keys, etc.)
• Use Edit Options to change question order, to
  view question, to delete or restore question
• Click Assemble Document button to print from
  your browser or to copy and paste into word
  processing applications

46
Printing Questions

• Print as usual from your browser or
• Copy the Print Folder contents into newer word
  processors (e.g., Microsoft Word)
• Go to Edit menu in your browser and choose
  Select All
• Select Copy from the Edit menu
• Open word processor and select Paste from the
  Edit menu (Save new file and edit as needed)

47
How to Use NAEP Questions

• Improve Students Abilities for Self-Assessment
• Teach Reasoning Skills/Strategies
• Gain Diagnostic Information/Analyze Mathematical
  Patterns of Error
• Teach Test-Taking Strategies and Use State
  Performance Data and NAEP Questions to Focus
  Instruction
• Springboard for Classroom Activities