Art Graesser

1
Assessments of Reading at Deeper Levels of
Comprehension

• Art Graesser
• Department of Psychology and the Institute for
  Intelligent Systems

2
Main Messages

• We need assessments of reading comprehension that
  systematically tap deeper levels of
  comprehension.
• Good deep measures integrate reading
  skills/strategies with world knowledge and
  nonlinguistic components.

3
Levels of Language Discourse

• Graphemes and phonemes
• Morphemes and words
• Syntactic composition
• Linguistic style and dialect
• Explicit propositions
• Referents of referring expressions
• Common ground
• Discourse focus versus presuppositions
• Situation models inferences
• Embedded dialog
• Configuration of multiple agents
• Genre, registers, rhetorical structures
• Plot configurations
• Local and global coherence
• Point of message -- theme
• Goals and attitude of author

• Five major categories (Kintsch, 1998 Graesser,
  Millis, Zwaan, 1997)
• Surface code
• Propositional textbase
• Situation model
• Text genre and rhetorical structure
• Pragmatic communication

4

• What do we mean by deep comprehension?

5
Cognitive Processes Bloom (1956)
Recognition
Evaluation
Recall
Synthesis
Comprehension (Inference)
Application
Analysis
6
Abstractness of Content (Mosenthal, 1996)

1. Most concrete concrete person, action, thing
2. Highly concrete observable attributes (action,
   types, attributes, amounts)
3. Intermediate Procedures, goals, and manner
4. Highly abstract cause, effect, reason, evidence
5. Most abstract inferred theories themes

7
Good Questions that Tap Pragmatic Communication
(Questioning the Author , Beck, McKeown,
Hamilton, Kucan, 1997)

• What is the author trying to tell you?
• Why is the author telling you that?
• Does the author say it clearly?
• How could the author have said things more
  clearly?
• What would you say instead?

8
Domain Knowledge Wisher Graesser (2007)
People (Agents)
Other
Taxonomies
Causal Networks
Spatial Layout
Compositional Structure
Procedures Plans
9

• Deep-Level Reasoning Questions
• (Graesser and Person,1994)
• LEVEL 1 SIMPLE or SHALLOW
• 1. Verification Is X true or false? Did an
  event occur?
• 2. Disjunctive Is X, Y, or Z the case?
• 3. Concept completion Who? What? When?
  Where?
• 4. Example What is an example or instance of a
  category?
• LEVEL 2 INTERMEDIATE
• 5. Feature specification What qualitative
  properties does entity X have?
• 6. Quantification What is the
  value of a quantitative variable? How much?
• 7. Definition questions What does X mean?
• 8. Comparison How is X similar to Y? How is X
  different from Y?
• LEVEL 3 COMPLEX or DEEP
• 9. Interpretation What
  concept/claim can be inferred from a pattern of
  data?
• 10. Causal antecedent Why did an event occur?
• 11. Causal consequence What are the consequences
  of an event or state?

10
Analysis of 120 Multiple Choice Questions in
Cognitive Psychology

• 120 MCQ items of cognitive psychology textbooks
• 4 test banks
• 30 items per textbook
• Items coded on the level in Graesser Person
  taxonomy.

11
Challenges

• How can we promote more diversity in the
  landscape of questions?
• How can we encourage deeper questions?
• How do we change
• Textbook writers
• Professors teachers
• Students

12
Cognitive and Behavioral Tests in Experimental
Psychology

• OFF-LINE
• Recall
• Summarization
• Retrospective think aloud
• Recognition tests
• Cloze procedure
• Multiple choice tests
• Sentence verification
• Question asking answering
• Response signal paradigm
• Ratings
• Word sorting

• ON-LINE
• Immediate think aloud
• Self-paced reading times
• Eye tracking
• Word naming latencies
• Lexical decision latencies
• RSVP-SOA
• Physiological recordings
• fMRI
• Evoked potential

13
Open-ended Questions

• The sun exerts a gravitational force on the earth
  as the earth moves in its orbit around the sun.
  Does the earth pull equally on the sun? Explain
  why?

EXPECTATIONS The sun exerts a gravitational force
on the earth. The earth exerts a gravitational
force on the sun. The two forces are a third-law
pair. The magnitudes of the two forces are the
same. MISCONCEPTIONS Only the larger object
exerts a force. The force of earth on sun is
less than that of the sun on earth.
14
Virtues of the Multiple Choice Format

• Familiar to the educational enterprise
• School systems
• Textbook companies
• Psychometric community
• Easy to score
• Objectively and systematically scored
• Distracters add complexity

15
Analytical Schemes

• Categories of distractors
• Near miss with pedagogical point (versus
  obscurity)
• Thematic versus unrelated distracters
• Relation to associated text
• Local versus global
• Explicit versus inference
• Main rhetorical content versus details
• Qualitative causal analysis
• How does a change in A affect B?
• Increase, decrease, no change, versus
  indeterminate

16
What happens to the pins when the key is turned
to unlock the door? A) they drop B) they rise
C) they remain stationary (Graesser Olde,
2003)
17
Closing Comments

• Rand Reading Study Group (Snow, 2002) emphasized
  comprehension
• Text
• Reader
• Activities
• Sociocultural context
• Training effective strategies of reading
  comprehension requires analysis of world
  knowledge and deeper levels of processing (edited
  by McNamara, 2007)

18

• ?