Title: Item Development Training for the Kentucky Core Content Test
1Item Development Training for the Kentucky Core
Content Test
- Adapted from WestEd Training for the
Kentucky Department of Education - June 2005
- Clarion HotelLouisville
2- Rhonda L. Sims, Director
- Division of Assessment Support
- Office of Assessment and Accountability
- 502-564-4394
- rhonda.sims_at_education.ky.gov
3Setting the Context
- State Assessment
- vs
- Continuous Classroom Assessment
- Changes Beginning in 2007
- New Test Design
- Depth of Knowledge Addition
- Refined Core Content for Assessment
4Test Development Process
- 1 For reading, Bias Committee review of pretest
passages - 2 Pretest item development by CACs
- 3 Editorial review of pretest items by WestEd
- 4 Bias Committee review of pretest items
- 5 Review and selection of pretest items for forms
by KDE and WestEd - 6 Layout and production of forms
- 7 Field test
- 8 Live administration
5Your Role as a CAC Member
- You are here representing your subject and grade
level. - You are here representing the entire Commonwealth
of Kentucky. - You need to consider the diversity of Kentucky
when developing items. - You must maintain confidentiality of item
content.
6Key Resources for Development
- Academic Expectations
- Program of Studies
- Core Content for Assessment
- Test Blueprint
- Kentucky Performance Level Descriptions
- Content resources
- These three play a key role in item development.
7Test Item Development
- General item-writing guidelines
- General guidelines regarding bias/sensitivity
issues - Guidelines for developing multiple-choice items
- Guidelines for Developing open-response items
8General Item-Writing Guidelines
- Alignment to Standards
- The match to Core Content for Assessment is
essential to the validity of the KCCT. - Content Relevancy
- Make sure the item assesses important knowledge
or skills identified by standards in the Kentucky
Core Content for Assessment.
9Sample Items
- The Preamble and the Articles of
- Confederation were written in which
- year?
- A. 1697
- B. 1765
- C. 1787
- D. 1865
10- Which document describes the
- purpose of the U.S. Constitution?
- A. Preamble
- B. Bill of Rights
- C. Articles of Confederation
- D. Declaration of Independence
11General Item-Writing Guidelines
- Grade-Level Appropriateness
- Make sure the item reflects the Content Standards
and Objectives at the appropriate grade level. - Basic vs. Technical Vocabulary
- Use simple, basic vocabulary instead of technical
vocabulary unless you are assessing the students
knowledge of the meaning of the technical
word/phrase.
12Sample Item
- Science, Grade 4
- When you plant a seed, the roots grow downward.
This is called geotropism. Which factor is
responsible for geotropism? - vs.
- When you plant a seed, the roots grow
downward, and the stem grows upward. Which
factor is responsible for the roots growing
downward?
13General Item-Writing Guidelines
- Grade Appropriateness Vocabulary
- Use grade-appropriate vocabulary as much as
possible. - Essential Information Only
14General Item-Writing Guidelines
- Clear Correct and Understandable Graphics
- Include clear, correct, easily understood
graphics as required by the item. - Bias and Sensitivity
- Consider Bias and Sensitivity Guidelines when
drafting items.
15- General Guidelines Regarding Bias/Sensitivity
Issues - Equal Opportunity and Access - The content and
language of the items should provide all students
with a fair opportunity to demonstrate what they
know, regardless of their - race
- ethnicity
- gender
- religion
- disability
- socioeconomic status, or
- region in which they live.
16General Guidelines Regarding Bias/Sensitivity
Issues
- Appropriate Portrayal of Groups - The subjects,
issues, and/or themes addressed by the items must
be approached in a way that does not demean,
offend, or inaccurately portray any religious,
ethnic, cultural, gender, or social group.
17General Guidelines Regarding Bias/Sensitivity
Issues
- Protection of Privacy - The content of the items
cannot intrude on the privacy of the values or
beliefs of students or their families.
18- General Guidelines Regarding Bias/Sensitivity
Issues - Bias should be avoided in
- item stems and prompts
- Response options and item directions
- graphics
- reading selections
- The emphasis should be on developing items that
reflect academic instruction versus life
experiences.
19Guidelines for Handling Sensitive Issues in
Assessment Development
- Issues to be avoided
- ? Child Abuse ? Suicide
- ? Incest ? Questioning Parental Authority
- ? Rape ? Occult
- ? Sex/Sexuality ? Divorce
- ? Sexual Preference or Orientation
20Guidelines for Handling Sensitive Issues in
Assessment Development
Issues to be used only in a broad context
- ? Birth Control
- ? Abortion
- ? Sexually Transmitted Diseases
- ? Witches/Witchcraft (Historical Context)
- ? Religious Holidays
21Guidelines for Handling Sensitive Issues in
Assessment Development
- ? Death ? Family Issues
- ? Guns/Gun Control ? Drugs/Alcohol/Tobacco
- ? Homelessness ? Murder
- ? Religion ? Pregnancy
- ? Creation/Evolution ? Violence
- ? Racism/Sexism/Ageism
- ? Use of Animals/Animal Rights
Issues which may or may not be suitable
22- Guidelines for the Development of
- Multiple-Choice Items
23Advantages of Multiple-Choice Items
- Can be used to measure a wide variety of learning
outcomes - Permit wide sampling and broad coverage of a
content domain - Are reliable and efficient to score
- Can provide useful diagnostic information about
the learning of individual students or groups of
students
24Disadvantages of Multiple-Choice Items
- Multiple-choice items are difficult to write.
- Multiple-choice items are not well suited for
measuring certain types of skills (e.g., the
ability to organize and express ideas in
writing). - Performance on multiple-choice items can be
influenced by student characteristics unrelated
to the subject of measurement, such as reading
ability and test-wiseness.
25Multiple-Choice Item Guidelines
- Be sure there is only one right answer.
- State the item stems in positive terms (if
possible). - Avoid using negatives in both the item stem and
the answer choices, double negatives are
confusing.
26- Multiple-Choice Item Guidelines
- Present a single, clear, complete problem or
question in the stem of the item. - Avoid the use of absolute terms in (always,
never, all, none, only).
27Multiple-Choice Item Guidelines
- Whenever possible, avoid answer choices that are
mutually exclusive opposites (e.g.,
fiction/nonfiction, living/non-living). When
such opposites are used, a students chance of
getting the item correct becomes 1 in 2 versus 1
in 4.
28When writing response options
- Make the incorrect response options plausible.
- Develop parallel response options. Options
should be parallel with respect to content,
structure and length.
29When writing response options
- Common misconceptions or errors of students are
good response options to include in an item. - Use other concepts or terms from the
grade-appropriate Core Content for Assessment.
30When writing response options
- Avoid humorous or nonsensical response options.
- Avoid using the options all of the above and
none of the above.
31When writing response options
- Logically order the response options. Numbers
should be listed in ascending or descending
order. Unless testing sequence of events from a
passage, list options in the order in which they
appear.
32 Training Examples for Multiple-Choice Items
33Do all four response options make sense? Each
response option should make sense both
grammatically and with respect to content.
34- In order to grow and flourish, pumpkins need lots
of - A. milk, eggs, and molasses.
- B. sun, water, and space.
- C. gourds, melons, and cucumbers.
- D. cream, eggs, and lots of sugar and
spices.
35Is the vocabulary used in the stem and response
options grade-level appropriate? It is very
important that the wording used in the stem and
response options be grade-level appropriate,
otherwise the item may be assessing a students
knowledge of vocabulary rather than content
knowledge.
36What is the one way heat can move from one object
to another? A. acceleration B.
insulation C. adaptation D.
conduction
37Is there one and only one right answer? Each
item must have only one right answer.
38 According to the passage, where do most home
accidents occur?
A. in the kitchen B. on the cutting
boards C. on the stove-top burners D. in
hot ovens
39Is there any inadvertent cueing going on in the
item? Sometimes items are written in such a way
that one of the response options contains an
important word or phrase from the stem.
40A style of architecture that uses the Roman arch,
thick walls, and dimly lit interiors is called
A. Gothic. B. modern. C.
Romanesque. D. Byzantine.
41Does one response option stand out from the
others in any way? Some items are written so
that one of the response options clearly stands
out from the other response options. There are
many ways that a response option may stand out
from the other options. One way is if one of the
options begins with a different word than the
other three options.
42In The Prince, Renaissance author, Machiavelli,
instructs the ruling monarchy in methods of
statesmanship and argues
A. for reconciliation of faith and reason.
B. that the ends justify the means. C. for
the rise of the common man. D. for allowing
freedom of thought.
43Another way that an option can stand out is in
length. If an option is much longer or shorter
than the other three options, it stands out.
44Some students want to build a pond near their
schoolyard. This change will MOST LIKELY
- A. increase the frog population but be
- harmful to the grass in the area.
- B. be harmful to both frogs and grass.
- C. not affect the plants and animals in
- the area.
- D. cause most animals to leave the area.
45- An option can stand out because it is the only
option that contains a negative or a negative
connotation. Response options need to be
balanced with respect to the use of negatives, so
that one option does not stand out.
46- Which is a reason Mike went to the park?
- He wanted to play with his friends.
- He did not want to go to school.
- He liked playing on the swings.
- He hoped to find his notebook.
47- Which is a reason Mike went to the park?
- He hoped to find his notebook.
- He did not want to go to school.
- He wanted to play with his friends.
- He did not want to stay home alone.
48- Janice can best be described as
- angry.
- mean.
- rude.
- caring.
49- Janice can best be described as
- angry.
- caring.
- mean.
- happy.
50Is the item format an appropriate one? Items
should not be written in certain formats. For
example, fill in the blank format should be
avoided, as should the analogy format. Analogies
in particular, are very difficult for many
students.
51Color is to art as _________ is to music. A.
melody B. rhythm C. dynamics D.
timbre
52Is there any extraneous information in the stem?
The stem of an item should only include
information that is essential to answering the
item. For example, in the item below, the first
sentence is not essential to the item.
53Corn is one of the largest crops grown in the
United States. There are 2,500,000,000 bushels
grown in the U.S. each year. If corn production
is decreased by 50 for each year, what will be
the U.S. corn production in 4 years?
54Does the item stem include all information
essential to answering the item? Just as some
items include too much information, other items
include too little. It is very important for an
item stem to include all of the information
necessary to answer the item or necessary for the
answer to be correct.
55To conduct an experiment Mike pulls marbles from
a bag. Each time he pulls a marble he returns it
to the bag. He does this 10 times. His results
are shown in the table below. There are 100
marbles in the bag. Theoretically, how many
marbles are in the bag. Red / Blue
A. 60 red, 40 blue B. 50 red, 50 blue C.
70 red, 30 blue D. 30 red, 70 blue
56- Activity
- Review of Multiple Choice
57Rumor Control Open Response
58OAAs Favorite OR Myths
- Restating the question is mandatory and will
earn the student one point. - Answers must be in paragraph form OR a graphic
organizer should be done on the response page. - Three or more examples must always be given.
- Doing more than required will score a 4.
- Scorers only have 30 seconds to score each
response. - Released items are BAD items that have been
thrown out of the test.
59- Guidelines for the Development of
- Open-Response Items
60Advantages of Open-Response Items
- Open-response items allow for more depth of
knowledge to be demonstrated than in
multiple-choice items. - Students can be asked to demonstrate more complex
cognitive behaviors such as comparing, relating,
analyzing, inferring, concluding, predicting,
generalizing, solving and/or applying.
61Disadvantages of Open-Response Items
- Open-response items are more difficult and more
expensive to score. - Because of their cost, the relative number of
open-response items is significantly less than
multiple choice. - Effectiveness of open-response items is based on
the scoring guide and answer information provided.
62Developing Open-Response Items
- Establishing the Prompt
- Designing Directions
63Establishing the Prompt
- What is the Core Content to be assessed?
- What specific prompt is appropriate to provide
context for the students to demonstrate their
knowledge? - Is the prompt age-appropriate?
- Does it include extraneous or misleading
information? -
- Even if the item has a graphic, there still must
be a prompt (1 or 2 sentences) that describes or
provides information related to the graphic
and/or item directions.
64Designing Directions
- Do the directions clearly reflect knowledge
and/or skills from the Core Content? - Do the directions specify what students must do?
- Do the directions provide students the
opportunity to reflect Proficient performance? - Are the directions clearly formulated?
- Is there a clear link between the prompt and the
directions? - Is the wording clear and focused?
- Is the task achievable?
- Is it answerable within one page?
65Make sure that the task is actually achievable.
- If the item asks students to read a passage or
examine a graphic and then give three ways or
explain two reasons based on the material
given, make sure there are three ways or two
reasons found in the material provided.
66- As a general rule, if students are asked to
generate a certain number of ideas on their own
without benefit of a passage or graphic, then
there should be at least twice that number of
possible answers. You will be asked to supply
the possible answers.
67Make sure the directions are complete and
specify what is desired from the students.
- If you want the students to provide examples, the
directions to the students should tell them to
provide examples and how many. - If you want the students to identify information,
do not ask them to discuss, describe or explain.
68Suggestions for Item Format
- Present the prompt in paragraph form.
- Use bullets to emphasize the details in the
prompt. - If the students are required to respond to
multiple parts of a question, label each part
separately (a, b, c).
69- Sample Item
- Martin said, I am thinking of a whole number
between 100 and 300.
- The number is divisible by three but not by 9.
- The ones digit is the sum of the hundreds digit
- and the tens digit.
- Show why 153 cannot be Martins number.
- Find all the numbers that match Martins clues.
- Show all your work.
- Write one more clue that would limit the answer
in part b to one and only one correct number.
70Types of Open-Response Items
- Scaffolded Questions
- Single Dimension/Component
- Student Choice Topics/Options Provided
- Response to Provided Information
71Scaffolded
- Sequences increasingly more difficult/complex
tasks - Success on one part would likely mean there was
success on previous parts - Multipart (simple to complex)
72Mathematics Example
- Corina was investigating information about
natural wonders of the world. - She found that Mt. Everest is the highest
mountain in the world. It is 29,028 feet ABOVE
sea level. - She found that the Marianas Trench in the Pacific
Ocean is the lowest point on Earth. It is 35,840
feet BELOW sea level.
73Mathematics Example (cont.)
- a. If Corina could throw a rock from the
- top of Mt. Everest to the bottom of the
Marianas Trench, how many feet would the rock
fall? - b. Draw a diagram and explain your answer for
part a.
74Single Dimension/Component
- Straightforward Question
- Draw a conclusion or take a position, then
support it - Explain a phenomenon or describe procedures
75Social Studies Example
- Many Kentucky cities are located near large
rivers. - Describe three important advantages that the
rivers provide these cities. - Explain why each advantage you described in part
a is important.
76Student Choice Topic/Options Provided
- Choose from the provided options
- Complete directions as specified
77Science Example
- Some of Earths materials are listed below
- Soil
- Water
- Gases of the atmosphere
- Choose TWO materials from the list. Explain how
a PLANT uses each of these materials to live. - Choose TWO materials from the list. Explain how
an ANIMAL uses each of these materials to live.
78Response To Provided Information
- Response to provided information such as data,
readings, graphics
79Reading Example
- In the story First Light, Matthew woke up in
another time period, the 1850s. - Describe FOUR things Matthew discovered that were
different from what he was used to in his present
life. - Explain how each of those differences affected
him. Use information from the story to support
your answer.
80When drafting open-response items
- Draft items on scratch paper.
- Brainstorm examples of information that would be
included in an acceptable response. - Exchange items and have someone else answer it.
- Compare response to what you expected.
- Evaluate the response to be sure that it is
possible to distinguish four levels of response
(0 4, Blank).
81When drafting open-response items
- 6. Revise item as needed.
- 7. Draft on chart paper.
- 8. Critique each item as a groupfor the match to
Core Content, grade-level appropriateness,
vocabulary, opportunity to demonstrate proficient
performance and whether answerable within the
one-page limit.
82Training Examples for Open-Response Items
83Is the item specific enough? Sometimes items are
written that are not specific enough. Such items
are usually difficult to score and unfair to
students. The item shown below is a good example
of an item that lacks specificity and would
probably pose a problem for students.
84 When Mikes class was studying insects, the
teacher asked each student to catch an insect and
keep it alive for a few weeks in a plastic jar.
- Name an insect that you think Mike should catch
and keep in the jar. - Draw a picture of the jar with all the features
that will be necessary for the insect to live for
a few weeks. Make sure that you labeled all
these features. - After Mike has placed his insect in the jar that
you you have drawn, what must he do to make sure
that the insect lives.
85Does the item stem inadvertently cue students to
one or more answers? Just as with multiple
choice items, it is important to make sure that
the stem of an open-response item does not cue
students to the answer(s).
86 In the article, the author writes, Pumpkins are
more than just symbols of autumn holidays. They
are thousands of years old and the stars of
folklore, fairy tales, and famous feasts.
- Describe three ways people used or are now using
pumpkins. - Explain why each way was or is important.
87- Activity
- Review of Open Response