Assessing Informational Text Comprehension

1
Assessing Informational Text Comprehension

• Nell K. Duke, Katherine R. Hilden, Alison K.
  Billman
• Michigan State University
• Literacy Achievement Research Center

2
Other Project Collaborators

• Juliet Halladay
• Yu Fang
• Yeow Meng Thum
• Yonghan Park
• Shenglan Zhang
• Angela Tanis
• Dipendra Subedi
• Michigan State University
• This project is supported by a grant from the
  Carnegie Corporation of New York and by the
  Literacy Achievement Research Center at Michigan
  State University.

3
Some Assumptions About Reading Comprehension
Development

• The foundations of reading comprehension
  development are laid long before children can
  actually read themselves.
• Reading comprehension is related to, but not the
  same as, listening comprehension.
• Reading comprehension is not automatic even given
  strong word recognition.
• Many different skills, dispositions, and bodies
  of knowledge are entailed in reading
  comprehension.
• Reading comprehension ability varies by many
  factors including by child and by type of text.

4
Some Existing Tools for Assessing Informational
Text Comprehension

• Think alouds or verbal protocols
• Can be used with a variety of texts
• Can provide insights into what readers are and
  are not thinking about when they are reading
  (e.g., level of activity, strategy use,
  connections)
• Suitability for young children still being
  investigated
• Procedures, reliability, etc. not entirely
  established
• Summaries or retellings
• Can provide insights into what children remember
  and view as important and their structural
  knowledge
• A skill in itself
• Procedures, reliability, etc. not established

5
Some Existing Tools for Assessing Informational
Text Comprehension

• Informal reading inventories (passages
  questions)
• Can estimate a childs comprehension level some
  have expository text passages
• Can indicate whether children have relatives
  strengths versus weaknesses in literal versus
  inferential questions
• Procedures established but leveling often
  questionable and reliability usually not
  established
• Self-evaluation checklists (e.g., When I read I.
  . .)
• Provide insights into childrens view of their
  comprehension
• Depends on childrens judgments
• Validity and reliability not established

6
Some Existing Tools for Assessing Informational
Text Comprehension

• Norm-Referenced Tests of Comprehension
• Usually do not distinguish informational
  comprehension from comprehension of other text
  types
• Usually cannot fully separate comprehension from
  word recognition
• Designed to place students on a normal curve
• Generally not very diagnostic
• Procedures and reliability established
• Others

7
Some New Tools for Assessing Informational Text
Comprehension

• We have been working on developing three new
  tools for assessing informational text
  comprehension in the primary grades.
• We want these tools to be useful to classroom
  teachers, reading specialists, and researchers.
• We want these tools to encourage more attention
  to informational text in research and practice.
• We want these tools to identify aspects of
  informational text comprehension worthy of
  assessment and instruction.

8
About the Assessments We Are Developing

• The assessments are designed to measure five
  specific dimensions of infotext comprehension
• Comprehension Strategy Use (CS) (Specifically,
  activating prior knowledge, predicting,
  inferring, summarizing)
• Knowledge of Informational Text Features (TF)
• Comprehension of Graphics in the Context of Text
  (GCT)
• Vocabulary Knowledge Knowledge of Tier II
  vocabulary for informational text (VK)
• Vocabulary Strategies Ability to ascertain Tier
  III word meaning from context (VS)

9
Comprehension Strategy Use

• There are differences in good and poor readers
  with respect to strategy use (e.g., Duke,
  Pressley, Hilden, 2004).
• Degree of strategy use related to comprehension
  achievement.
• The relationship seems to be causal in that
  instruction in comprehension strategies has been
  shown to improve comprehension (e.g., Duke
  Pearson, 2002).
• Included inferencing although that is often
  spontaneous rather than strategic in the
  traditional sense.

10
Knowledge of Informational Text Features

• There are differences in good and poor readers
  with respect to knowledge some text features
  text structure, at least (e.g., Dickson, Simmons,
  Kameenui, 1995).
• There is some causation at work in that text
  structure instruction can improve comprehension
  (e.g., Dickson, Simmons, Kameenui, 1995), as
  can instruction in searching (using index,
  headings, etc.) (Symons, MacLatchy-Gaudet, Stone,
  Reynolds, 2001).
• Research has not yet established differences in
  good and poor readers, or causation, for a number
  of other informational text features.

11
Comprehension of Graphics in the Context of Text

• It appears that illustrations can have a
  facilitative effect on comprehension for at least
  some readers, although this does not seem to
  divide neatly along lines of good versus poor
  readers (see Gyselinck Tardieu, 1999, for a
  review).
• To our knowledge, it has not yet been shown
  whether informational text comprehension can be
  improved by instruction in building meaning
  through illustrations as well as text.
• For now, we are assuming that the ability to
  comprehend graphics in the context of text is
  important and amenable to instruction.

12
Vocabulary Knowledge

• There are differences in good and poor readers
  with respect to vocabulary knowledge.
• Vocabulary knowledge is related to comprehension
  achievement (Blachowicz Fisher, 2000).
• The relationship seems to be causal in that
  instruction in vocabulary has been shown to
  improve comprehension (e.g., Baumann, Edwards,
  Boland, Olejnick, Kameenui, 2003).
• We focus on knowledge of high-utility vocabulary
  (similar to what Beck, McKeown, and Kucan, 2002
  call Tier II vocabulary in reference to narrative
  text) words (for science text) like compare,
  describe, observe, kinds, tools, cycle.

13
Vocabulary Strategies

• Research is mixed as to whether good and poor
  readers differ in their ability to infer word
  meaning from context, or the degree to which
  vocabulary strategy use is related to
  comprehension achievement.
• In at least some studies, instruction in
  vocabulary strategies has been shown to improve
  comprehension (e.g., Baumann, Edwards, Boland,
  Olejnick, Kameenui, 2003).
• We focus on the ability to ascertain from context
  the meaning of fairly low-incidence or Tier III
  words.

14
Three Assessments We are Developing

• Concepts of Comprehension Assessment (COCA), for
  informational text -- this is the one we will
  focus on today
• Strategic Cloze Assessment, for informational
  text
• Performance Task, for informational text

15
COCA Quality

• There are three forms of the COCA Frogs,
  Butterflies, and Dogs.
• Based on administration to first and second grade
  students at various points in time in the year
• Overall reliabilities are
• Frogs .833
• Monarchs .802
• Dogs .745
• Scores generally higher in March than October and
  generally higher for second grade than first.
• Unsure yet of relationship between scores and
  instruction.

16
COCA Quality

• Frogs Reliability by Factors
• CS 0.525
• TF 0.784
• GCT 0.730
• VK 0.837
• VS 0.719

17
COCA Quality

• Monarchs Reliability by Factors
• CS 0.757
• TF 0.789
• GCT 0.583
• VK 0.750
• VS 0. 754

18
COCA Administration Procedures

• The COCA is individually administered.
• Assessment sessions run 15-20 minutes per child.
• Text and prompts are scripted to facilitate
  consistent administration.
• Student responses are recorded directly on the
  score sheet to facilitate scoring.

19
Administration Pointers

• It is important to be consistent across
  administrations.
• It is important to read as naturally as possible
  with good inflection.
• It is important to break between the text and the
  prompt or question.

20
Some Next Steps

• Examine relationship of COCA scores to
  instruction
• Examine use of COCA with end-of-third grade
  children
• Make COCA available in the public domain
  (msularc.org)
• Investigate predictive validity of COCA
• Continue to examine the quality of the other two
  assessments we have been developing.