Standard Setting

1
Standard Setting Different names for the same
thing Standard Passing Score Cut Score Cutoff
Score Mastery Level Bench Mark
2
Standards exist for as long as there are
tests Canada 50 China 60 America
70 France 2/3
3
Examinee-Centered Methods Borderline Group
Method Contrasting Groups Method Problem How
to identify the examinees. Test-Centered
Methods Nedelsky Method (1954) Angoff Method
(1971) IDEA Method (2004) Problem Judgmental
errors.
4
Test-Centered Methods Conceptualization of
examinee competency Judgmental item
analysis Aggregating item difficulty estimates
5
Training of Judges Conceptualization of minimum
competency The use of a standard-setting
method Intrajudge inconsistency Iterative
process Documentation
6
What do experts say? "We have come to realize
that there is no objectively correct way to set
standards. But we have also come to realize that
there is nothing wrong with using judgments
appropriately." (Zieky, 1995, p.5) "Determinatio
n of a minimum acceptable performance always
involves some rather arbitrary and not wholly
satisfactory decisions." (Ebel, 1972, p.492)
7
"Researchers agree with Glass (1978) that
standards are all arbitrary. But they reject the
notion that being arbitrary, in the sense of
being judgmental, is a fatal flaw." (Zieky, 1994,
p28) "If competence is a continuous variable,
there is clearly no point on the continuum that
would separate students into the competent and
the incompetent. A right answer does not exist,
except perhaps in the minds of those providing
the judgments." (Jaeger, 1989, p.492)
8
1. All standard-setting is judgmental 2.
Standard-setting leads to errors of
classification 3. Standard-setting is and will
remain controversial 4. There is no purely
absolute standard. 5. There is no one right
method 6. Choosing judges is more important than
choosing methods 7. Standard-setting is a
process
9
Failed Standard-Setting Exercises Due to Legal
Matters Tenured teachers can not be
decertified. Contracted teachers can not be
decertified. Candidates for becoming teachers
can be decertified Due to Psychometric
Matters Practice analysis failed to support
job-relatedness of the test. Teachers concern
about the objective of the test was not
addressed. Items did not reflect judgment of
content committee. Teachers were excluded from
the standard-setting processes. Changing
cutscore without justifications.
10
Defensible Standard Setting Steps Subject
matter experts are asked to review test
items. Sensitivity review to check biases
against certain groups. Documentation of the
standard setting processes Description of
subject matter experts Selection criteria and
procedures Standard-setting methods and
justifications Training procedures Independent
evidence that the cutscore is reasonable. Any
indices of reliability, item analysis
information, distractor analysis
Intrajudge and interjudge consistency
evidence, e.g., split half
11
(No Transcript)
12
Steps and Procedures in Developing an Achievement
Test
1. Define domain content
Intelligence tests and theories, 20
Personality tests, 20
Item characteristics, 10
Reliability, 20
Validity, 15
Test development, 15
Table of specifications
High Low Total
Intelligence tests and theories 5 15 20
Personality tests, 5 15 20
Item characteristics, 5 5 10
Reliability, 15 5 20
Validity, 10 5 15
Test development, 5 10 15
13
2. Write Items Normally done by subject
teachers.

• 3. Item Analysis
• Item variance
• Item difficulty
• Item discrimination

4. Setting Standards

• 5. Test reliability
• Test-retest
• Parallel form
• Split half-internal consistency

• 6. Test Validity
• various validity issues

14
"Anyone who expects to discover the "real"
passing score by any of these approaches, or any
other approach, is doomed to disappointment, for
a "real" passing score does not exist to be
discovered. All any examining authority that
must set passing scores can hope for, and all any
of their examinees can ask, is that the basis for
defining the passing score be defined clearly,
and that the definition be as rational as
possible." (Ebel, 1972, p.496) "At a minimum,
standard-setting procedures should include a
balancing of absolute judgments and direct
attention to passing rates." (Shepard, 1980,
p.463)
15
Uses of Standards Exhortation To exhort,
encourage or urge the students, schools, and
public to exert more or different kinds of effort
to achieve established standards of
performance. Exemplification of Goals To
provide clear specifications of the achievement
levels that students are expected to
attain. Accountability In the U.S., schools not
students can obtain rewards or sanctions
depending on the progress achieved in meeting
performance standards. Certification of
Achievement and Mastery
16
"The thing that hath been, it is that which
shall be and that which is done is that which
shall be done and there is no new thing under
the sun." Ecclesiastes (19).