American Institutes for Research

1
Evaluating Assessments for the NRS

• American Institutes for Research
• February 2005

2
Student Assessment Process
Student
Adapted from D.M. Kenyon and C. Van Duzer, Valid,
Reliable, and Appropriate Assessments for Adult
English Language Learners, November 2003
http//www.cal.org/caela/accountability
3
Need for Standardized Assessment Review Process

• Promotes compatibility of assessment information
  across states
• Standard process and criteria for determining use
  to measure educational gain
• Without it, inconsistent approval of assessments
  across states

4
Assessment Review Process

• Required review procedure for states and at
  federal level
• All assessments to be reviewed
• Plan to formally regulate requirements, to take
  effect in 2006
• Guided by Higher Education Ability to Benefit
  approach
• To be published in new NRS Implementation
  Guidelines (draft in handouts)

5
Assessment Review Process (2)

• Step 1 ApplicationState or test publisher
  requests review and submits
• Source materials (e.g., manuals, report)
• Reports with psychometric evidence (e.g.,
  validity, reliability)
• Step 2 Rating on Psychometric Criteria
• Using NRS criteria on test requirements

6
Assessment Review Process (3)

• Step 3 Formal Review
• Panel of 2 or 3 experts independently reviews
  assessment and makes recommendation
• Step 4 Documentation
• Formal report and recommendation to OVAE

7
Questions, Discussion

• Questions?

8
NRS Assessment Database

• Access to an online assessment database through
  www.NRSWeb.org
• Database will contain assessments that are
  appropriate for use in adult education programs
• Assessments will be approved for inclusion in the
  database by ED
• Additional assessments may be added after review
  and approval by ED

9
Content

• Database will include 4 main sections
• List of tests included in the database
• List of features to be included
• Results pages (for search and comparison
  functions)
• Definitions of terms

10
Assessments by Name/Title
11
Features to Be Included

• Title of Test (Acronym)
• Narrative Summaryshort statement
• Test Use and Purpose
• Intended Population
• Purpose (e.g., Placement, Diagnostic, Progress,
  Achievement/Proficiency)

12
Features to Be Included (contd.)

• Test Specifications and Features
• Test Content (skills measured)
• Range of Skill Levels
• Items and Subtests
• Alternate Test Forms
• Required/Recommended Training for Test
  Administrators
• Test Administration Procedures
• Scoring and Reporting
• Time Needed for Assessment

13
Features to Be Included (contd.)

• Ordering Information/Availability
• Publisher/Company/Source for Obtaining the Test
• Versions, Publication Date(s)
• Product Web Site
• Additional Evaluator Comments

14
Functions

• Two search methods
• Search for an assessment by name
• Search for an assessment according to specific
  features

15
Search Results by Test Name/Title
16
Comparison Function

• Compare up to three different assessments across
  features in a side-by-side table format

17
Comparison Chart of Two Tests
18
NRS Assessment Database

• Questions?
• Comments/Feedback