Psychometric Evaluation of the National Board of Professional Teaching Standards (NBPTS) Assessments

1
Psychometric Evaluation of the National Board of
Professional Teaching Standards (NBPTS)
Assessments

• Teresa L. Russell, Ph.D.
• 28 November 2006

2
Objectives

• Broad review to summarize and examine information
  across assessments
• In-depth review to focus on either
• A specific set of assessments or
• A feature common to all assessments

3
Broad Review Objectives

• Summarize the technical characteristics across
  assessments,
• Recommend assessments or specific features of the
  assessment development process to be covered in
  the in-depth analysis, and
• Summarize the processes used to develop content
  standards and cut scores.

4
Broad Review Objective 1Summarize Technical
Information

• Review the existing technical reports for the
  past three administration cycles.
• Summarize technical information such as
• Examinee volume levels,
• Assessment-specific information (e.g., revisions)
• Other technical information
• Total pass rates
• Subgroup pass rates
• Reliability estimates
• Exercise intercorrelations.

5
Broad Review Objective 2Recommend In-Depth
Review Approach

• Focus on assessments (e.g., ones flagged from the
  broad review or ones having some other
  characteristic) or
• Focus on features common to all assessments (e.g.
  reliability, disparate impact, portfolio/
  assessment center overlap)

6
Broad Review Objective 3 Summarize Processes

• Review materials containing historical
  information about the development and updating of
  the content standards (including efforts to
  minimize bias)
• Summarize the development and updating of the
  content standards in a form that is useful to the
  Committee.
• Review materials related to the process for
  setting performance standards (cut scores),
  including attention given to ensuring fairness.

7
In-Depth Review Objective

• If the focus is on assessments, the evaluation
  will include some or all of the following
  elements
• Purpose of the Assessment
• Description of the Assessment
• Competencies to be Assessed
• Content Development Process
• Standard Setting Process
• Psychometric Characteristics
• If features are the focus of the review, we will
  compare the features to professional standards
  and if available, published literature.