Item Development Training for the Kentucky Core Content Test

1
Item Development Training for the Kentucky Core
Content Test

• Adapted from WestEd Training for the
  Kentucky Department of Education
• June 2005
• Clarion HotelLouisville

2

• Rhonda L. Sims, Director
• Division of Assessment Support
• Office of Assessment and Accountability
• 502-564-4394
• rhonda.sims_at_education.ky.gov

3
Setting the Context

• State Assessment
• vs
• Continuous Classroom Assessment
• Changes Beginning in 2007
• New Test Design
• Depth of Knowledge Addition
• Refined Core Content for Assessment

4
Test Development Process

• 1 For reading, Bias Committee review of pretest
  passages
• 2 Pretest item development by CACs
• 3 Editorial review of pretest items by WestEd
• 4 Bias Committee review of pretest items
• 5 Review and selection of pretest items for forms
  by KDE and WestEd
• 6 Layout and production of forms
• 7 Field test
• 8 Live administration

5
Your Role as a CAC Member

• You are here representing your subject and grade
  level.
• You are here representing the entire Commonwealth
  of Kentucky.
• You need to consider the diversity of Kentucky
  when developing items.
• You must maintain confidentiality of item
  content.

6
Key Resources for Development

• Academic Expectations
• Program of Studies
• Core Content for Assessment
• Test Blueprint
• Kentucky Performance Level Descriptions
• Content resources
• These three play a key role in item development.

7
Test Item Development

• General item-writing guidelines
• General guidelines regarding bias/sensitivity
  issues
• Guidelines for developing multiple-choice items
• Guidelines for Developing open-response items

8
General Item-Writing Guidelines

• Alignment to Standards
• The match to Core Content for Assessment is
  essential to the validity of the KCCT.
• Content Relevancy
• Make sure the item assesses important knowledge
  or skills identified by standards in the Kentucky
  Core Content for Assessment.

9
Sample Items

• The Preamble and the Articles of
• Confederation were written in which
• year?
• A. 1697
• B. 1765
• C. 1787
• D. 1865

10

• Which document describes the
• purpose of the U.S. Constitution?
• A. Preamble
• B. Bill of Rights
• C. Articles of Confederation
• D. Declaration of Independence

11
General Item-Writing Guidelines

• Grade-Level Appropriateness
• Make sure the item reflects the Content Standards
  and Objectives at the appropriate grade level.
• Basic vs. Technical Vocabulary
• Use simple, basic vocabulary instead of technical
  vocabulary unless you are assessing the students
  knowledge of the meaning of the technical
  word/phrase.

12
Sample Item

• Science, Grade 4
• When you plant a seed, the roots grow downward.
  This is called geotropism. Which factor is
  responsible for geotropism?
• vs.
• When you plant a seed, the roots grow
  downward, and the stem grows upward. Which
  factor is responsible for the roots growing
  downward?

13
General Item-Writing Guidelines

• Grade Appropriateness Vocabulary
• Use grade-appropriate vocabulary as much as
  possible.
• Essential Information Only

14
General Item-Writing Guidelines

• Clear Correct and Understandable Graphics
• Include clear, correct, easily understood
  graphics as required by the item.
• Bias and Sensitivity
• Consider Bias and Sensitivity Guidelines when
  drafting items.

15

• General Guidelines Regarding Bias/Sensitivity
  Issues
• Equal Opportunity and Access - The content and
  language of the items should provide all students
  with a fair opportunity to demonstrate what they
  know, regardless of their
• race
• ethnicity
• gender
• religion
• disability
• socioeconomic status, or
• region in which they live.

16
General Guidelines Regarding Bias/Sensitivity
Issues

• Appropriate Portrayal of Groups - The subjects,
  issues, and/or themes addressed by the items must
  be approached in a way that does not demean,
  offend, or inaccurately portray any religious,
  ethnic, cultural, gender, or social group.

17
General Guidelines Regarding Bias/Sensitivity
Issues

• Protection of Privacy - The content of the items
  cannot intrude on the privacy of the values or
  beliefs of students or their families.

18

• General Guidelines Regarding Bias/Sensitivity
  Issues
• Bias should be avoided in
• item stems and prompts
• Response options and item directions
• graphics
• reading selections
• The emphasis should be on developing items that
  reflect academic instruction versus life
  experiences.

19
Guidelines for Handling Sensitive Issues in
Assessment Development

• Issues to be avoided
• ? Child Abuse ? Suicide
• ? Incest ? Questioning Parental Authority
• ? Rape ? Occult
• ? Sex/Sexuality ? Divorce
• ? Sexual Preference or Orientation

20
Guidelines for Handling Sensitive Issues in
Assessment Development
Issues to be used only in a broad context

• ? Birth Control
• ? Abortion
• ? Sexually Transmitted Diseases
• ? Witches/Witchcraft (Historical Context)
• ? Religious Holidays

21
Guidelines for Handling Sensitive Issues in
Assessment Development

• ? Death ? Family Issues
• ? Guns/Gun Control ? Drugs/Alcohol/Tobacco
• ? Homelessness ? Murder
• ? Religion ? Pregnancy
• ? Creation/Evolution ? Violence
• ? Racism/Sexism/Ageism
• ? Use of Animals/Animal Rights

Issues which may or may not be suitable
22

• Guidelines for the Development of
• Multiple-Choice Items

23
Advantages of Multiple-Choice Items

• Can be used to measure a wide variety of learning
  outcomes
• Permit wide sampling and broad coverage of a
  content domain
• Are reliable and efficient to score
• Can provide useful diagnostic information about
  the learning of individual students or groups of
  students

24
Disadvantages of Multiple-Choice Items

• Multiple-choice items are difficult to write.
• Multiple-choice items are not well suited for
  measuring certain types of skills (e.g., the
  ability to organize and express ideas in
  writing).
• Performance on multiple-choice items can be
  influenced by student characteristics unrelated
  to the subject of measurement, such as reading
  ability and test-wiseness.

25
Multiple-Choice Item Guidelines

• Be sure there is only one right answer.
• State the item stems in positive terms (if
  possible).
• Avoid using negatives in both the item stem and
  the answer choices, double negatives are
  confusing.

26

• Multiple-Choice Item Guidelines
• Present a single, clear, complete problem or
  question in the stem of the item.
• Avoid the use of absolute terms in (always,
  never, all, none, only).

27
Multiple-Choice Item Guidelines

• Whenever possible, avoid answer choices that are
  mutually exclusive opposites (e.g.,
  fiction/nonfiction, living/non-living). When
  such opposites are used, a students chance of
  getting the item correct becomes 1 in 2 versus 1
  in 4.

28
When writing response options

• Make the incorrect response options plausible.
• Develop parallel response options. Options
  should be parallel with respect to content,
  structure and length.

29
When writing response options

• Common misconceptions or errors of students are
  good response options to include in an item.
• Use other concepts or terms from the
  grade-appropriate Core Content for Assessment.

30
When writing response options

• Avoid humorous or nonsensical response options.
• Avoid using the options all of the above and
  none of the above.

31
When writing response options

• Logically order the response options. Numbers
  should be listed in ascending or descending
  order. Unless testing sequence of events from a
  passage, list options in the order in which they
  appear.

32
Training Examples for Multiple-Choice Items
33

Do all four response options make sense? Each
response option should make sense both
grammatically and with respect to content.
34

• In order to grow and flourish, pumpkins need lots
  of
• A. milk, eggs, and molasses.
• B. sun, water, and space.
• C. gourds, melons, and cucumbers.
• D. cream, eggs, and lots of sugar and
  spices.

35
Is the vocabulary used in the stem and response
options grade-level appropriate? It is very
important that the wording used in the stem and
response options be grade-level appropriate,
otherwise the item may be assessing a students
knowledge of vocabulary rather than content
knowledge.
36
What is the one way heat can move from one object
to another? A. acceleration B.
insulation C. adaptation D.
conduction
37
Is there one and only one right answer? Each
item must have only one right answer.
38
According to the passage, where do most home
accidents occur?
A. in the kitchen B. on the cutting
boards C. on the stove-top burners D. in
hot ovens
39
Is there any inadvertent cueing going on in the
item? Sometimes items are written in such a way
that one of the response options contains an
important word or phrase from the stem.
40
A style of architecture that uses the Roman arch,
thick walls, and dimly lit interiors is called
A. Gothic. B. modern. C.
Romanesque. D. Byzantine.
41
Does one response option stand out from the
others in any way? Some items are written so
that one of the response options clearly stands
out from the other response options. There are
many ways that a response option may stand out
from the other options. One way is if one of the
options begins with a different word than the
other three options.
42
In The Prince, Renaissance author, Machiavelli,
instructs the ruling monarchy in methods of
statesmanship and argues
A. for reconciliation of faith and reason.
B. that the ends justify the means. C. for
the rise of the common man. D. for allowing
freedom of thought.
43
Another way that an option can stand out is in
length. If an option is much longer or shorter
than the other three options, it stands out.
44
Some students want to build a pond near their
schoolyard. This change will MOST LIKELY

• A. increase the frog population but be
• harmful to the grass in the area.
• B. be harmful to both frogs and grass.
• C. not affect the plants and animals in
• the area.
• D. cause most animals to leave the area.

45

• An option can stand out because it is the only
  option that contains a negative or a negative
  connotation. Response options need to be
  balanced with respect to the use of negatives, so
  that one option does not stand out.

46

• Which is a reason Mike went to the park?
• He wanted to play with his friends.
• He did not want to go to school.
• He liked playing on the swings.
• He hoped to find his notebook.

47

• Which is a reason Mike went to the park?
• He hoped to find his notebook.
• He did not want to go to school.
• He wanted to play with his friends.
• He did not want to stay home alone.

48

• Janice can best be described as
• angry.
• mean.
• rude.
• caring.

49

• Janice can best be described as
• angry.
• caring.
• mean.
• happy.

50
Is the item format an appropriate one? Items
should not be written in certain formats. For
example, fill in the blank format should be
avoided, as should the analogy format. Analogies
in particular, are very difficult for many
students.
51
Color is to art as _________ is to music. A.
melody B. rhythm C. dynamics D.
timbre
52
Is there any extraneous information in the stem?
The stem of an item should only include
information that is essential to answering the
item. For example, in the item below, the first
sentence is not essential to the item.
53
Corn is one of the largest crops grown in the
United States. There are 2,500,000,000 bushels
grown in the U.S. each year. If corn production
is decreased by 50 for each year, what will be
the U.S. corn production in 4 years?
54
Does the item stem include all information
essential to answering the item? Just as some
items include too much information, other items
include too little. It is very important for an
item stem to include all of the information
necessary to answer the item or necessary for the
answer to be correct.
55
To conduct an experiment Mike pulls marbles from
a bag. Each time he pulls a marble he returns it
to the bag. He does this 10 times. His results
are shown in the table below. There are 100
marbles in the bag. Theoretically, how many
marbles are in the bag. Red / Blue
A. 60 red, 40 blue B. 50 red, 50 blue C.
70 red, 30 blue D. 30 red, 70 blue
56

• Activity
• Review of Multiple Choice

57
Rumor Control Open Response
58
OAAs Favorite OR Myths

• Restating the question is mandatory and will
  earn the student one point.
• Answers must be in paragraph form OR a graphic
  organizer should be done on the response page.
• Three or more examples must always be given.
• Doing more than required will score a 4.
• Scorers only have 30 seconds to score each
  response.
• Released items are BAD items that have been
  thrown out of the test.

59

• Guidelines for the Development of
• Open-Response Items

60
Advantages of Open-Response Items

• Open-response items allow for more depth of
  knowledge to be demonstrated than in
  multiple-choice items.
• Students can be asked to demonstrate more complex
  cognitive behaviors such as comparing, relating,
  analyzing, inferring, concluding, predicting,
  generalizing, solving and/or applying.

61
Disadvantages of Open-Response Items

• Open-response items are more difficult and more
  expensive to score.
• Because of their cost, the relative number of
  open-response items is significantly less than
  multiple choice.
• Effectiveness of open-response items is based on
  the scoring guide and answer information provided.

62
Developing Open-Response Items

• Establishing the Prompt
• Designing Directions

63
Establishing the Prompt

• What is the Core Content to be assessed?
• What specific prompt is appropriate to provide
  context for the students to demonstrate their
  knowledge?
• Is the prompt age-appropriate?
• Does it include extraneous or misleading
  information?
• Even if the item has a graphic, there still must
  be a prompt (1 or 2 sentences) that describes or
  provides information related to the graphic
  and/or item directions.

64
Designing Directions

• Do the directions clearly reflect knowledge
  and/or skills from the Core Content?
• Do the directions specify what students must do?
• Do the directions provide students the
  opportunity to reflect Proficient performance?
• Are the directions clearly formulated?
• Is there a clear link between the prompt and the
  directions?
• Is the wording clear and focused?
• Is the task achievable?
• Is it answerable within one page?

65
Make sure that the task is actually achievable.

• If the item asks students to read a passage or
  examine a graphic and then give three ways or
  explain two reasons based on the material
  given, make sure there are three ways or two
  reasons found in the material provided.

66

• As a general rule, if students are asked to
  generate a certain number of ideas on their own
  without benefit of a passage or graphic, then
  there should be at least twice that number of
  possible answers. You will be asked to supply
  the possible answers.

67
Make sure the directions are complete and
specify what is desired from the students.

• If you want the students to provide examples, the
  directions to the students should tell them to
  provide examples and how many.
• If you want the students to identify information,
  do not ask them to discuss, describe or explain.

68
Suggestions for Item Format

• Present the prompt in paragraph form.
• Use bullets to emphasize the details in the
  prompt.
• If the students are required to respond to
  multiple parts of a question, label each part
  separately (a, b, c).

69

• Sample Item
• Martin said, I am thinking of a whole number
  between 100 and 300.

• The number is divisible by three but not by 9.
• The ones digit is the sum of the hundreds digit
• and the tens digit.
• Show why 153 cannot be Martins number.
• Find all the numbers that match Martins clues.
• Show all your work.
• Write one more clue that would limit the answer
  in part b to one and only one correct number.

70
Types of Open-Response Items

• Scaffolded Questions
• Single Dimension/Component
• Student Choice Topics/Options Provided
• Response to Provided Information

71
Scaffolded

• Sequences increasingly more difficult/complex
  tasks
• Success on one part would likely mean there was
  success on previous parts
• Multipart (simple to complex)

72
Mathematics Example

• Corina was investigating information about
  natural wonders of the world.
• She found that Mt. Everest is the highest
  mountain in the world. It is 29,028 feet ABOVE
  sea level.
• She found that the Marianas Trench in the Pacific
  Ocean is the lowest point on Earth. It is 35,840
  feet BELOW sea level.

73
Mathematics Example (cont.)

• a. If Corina could throw a rock from the
• top of Mt. Everest to the bottom of the
  Marianas Trench, how many feet would the rock
  fall?
• b. Draw a diagram and explain your answer for
  part a.

74
Single Dimension/Component

• Straightforward Question
• Draw a conclusion or take a position, then
  support it
• Explain a phenomenon or describe procedures

75
Social Studies Example

• Many Kentucky cities are located near large
  rivers.
• Describe three important advantages that the
  rivers provide these cities.
• Explain why each advantage you described in part
  a is important.

76
Student Choice Topic/Options Provided

• Choose from the provided options
• Complete directions as specified

77
Science Example

• Some of Earths materials are listed below
• Soil
• Water
• Gases of the atmosphere
• Choose TWO materials from the list. Explain how
  a PLANT uses each of these materials to live.
• Choose TWO materials from the list. Explain how
  an ANIMAL uses each of these materials to live.
  

78
Response To Provided Information

• Response to provided information such as data,
  readings, graphics

79
Reading Example

• In the story First Light, Matthew woke up in
  another time period, the 1850s.
• Describe FOUR things Matthew discovered that were
  different from what he was used to in his present
  life.
• Explain how each of those differences affected
  him. Use information from the story to support
  your answer.

80
When drafting open-response items

• Draft items on scratch paper.
• Brainstorm examples of information that would be
  included in an acceptable response.
• Exchange items and have someone else answer it.
• Compare response to what you expected.
• Evaluate the response to be sure that it is
  possible to distinguish four levels of response
  (0 4, Blank).

81
When drafting open-response items

• 6. Revise item as needed.
• 7. Draft on chart paper.
• 8. Critique each item as a groupfor the match to
  Core Content, grade-level appropriateness,
  vocabulary, opportunity to demonstrate proficient
  performance and whether answerable within the
  one-page limit.

82
Training Examples for Open-Response Items
83
Is the item specific enough? Sometimes items are
written that are not specific enough. Such items
are usually difficult to score and unfair to
students. The item shown below is a good example
of an item that lacks specificity and would
probably pose a problem for students.
84
When Mikes class was studying insects, the
teacher asked each student to catch an insect and
keep it alive for a few weeks in a plastic jar.

• Name an insect that you think Mike should catch
  and keep in the jar.
• Draw a picture of the jar with all the features
  that will be necessary for the insect to live for
  a few weeks. Make sure that you labeled all
  these features.
• After Mike has placed his insect in the jar that
  you you have drawn, what must he do to make sure
  that the insect lives.

85
Does the item stem inadvertently cue students to
one or more answers? Just as with multiple
choice items, it is important to make sure that
the stem of an open-response item does not cue
students to the answer(s).
86
In the article, the author writes, Pumpkins are
more than just symbols of autumn holidays. They
are thousands of years old and the stars of
folklore, fairy tales, and famous feasts.

• Describe three ways people used or are now using
  pumpkins.
• Explain why each way was or is important.

87

• Activity
• Review of Open Response