Assessing Academic Literacy: The role of text in comprehending written language - PowerPoint PPT Presentation


PPT – Assessing Academic Literacy: The role of text in comprehending written language PowerPoint presentation | free to view - id: 835f3c-NWIzM


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation

Assessing Academic Literacy: The role of text in comprehending written language


Is it poor teaching after 8th grade? ... dialogue, symbolism, imagery, irony, figurative language author s craft sequence, cause and effect, ... – PowerPoint PPT presentation

Number of Views:244
Avg rating:3.0/5.0
Slides: 67
Provided by: Para182


Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: Assessing Academic Literacy: The role of text in comprehending written language

Assessing Academic Literacy The role of text in
comprehending written language
  • Barbara Foorman, Ph.D.
  • Florida Center for Reading Research
  • Florida State University

What are the Issues?
  • Academic literacy assumes grade-level
  • On the 2007 Reading NAEP, 33 below basic in G4
    26 below basic in G8.
  • For minorities, the below basic on the 2007
    Reading NAEP are 53 in G4 45 in G8 for
    Blacks 50 in G4 and 42 in G8 for Hispanics.
  • NCLB requires that students at-risk for reading
    disability receive intervention.

Goals for This Presentation
  • Explain relation of academic literacy to academic
  • Definitions of reading comprehension
  • Characteristics of text difficulty
  • Measuring text difficulty
  • Assessing academic literacy

Academic Language is at the Core of Literacy
  • because it allows literate people to discuss
    literary products previously referred to as
    extended discourse or decontextualized language.
  • because contextual cues and shared assumptions
    are minimized by explicitly encoding referents
    for pronouns, actions, and locations

13 higher- SES children (professional)
23 middle/lower- SES children (working class)
Cumulative Vocabulary words
6 welfare children
Age of child in months
Hart Risley, 1995
Language Experience
Estimated cumulative words addressed to child
Age of child in months
Hart Risley, 1995
Quality Teacher Talk(Snow et al., 2007)
  • Rare words
  • Ability to listen to children and to extend their
  • Tendency to engage children in cognitively
    challenging talk
  • Promotes emergent literacy vocabulary
    literacy success in middle grades

Home Schoolexperiences ages 3-6 Skills
developed ages 3-6 School performance
Understanding literacy
Kindergarten and first grade reading
Print focus
Conversational language
Instruction and Practice in reading
Extended discourse forms and nonfamiliar audiences
Reading comprehension In Grade 4
Decontextualized language
(Snow, 1991)
(Cunningham Stanovich, 1998, adapted from
Anderson, Wilson, Fielding,1988)
Is Literacy Enough? (Snow et al., 2007)
  • For adolescents, oral language and literacy
    skills need to be adequate, but also need
  • Caring adult(s) at home
  • Caring adults at school who provide guidance
    about how to meet goals (often need smaller
  • Minimal risk Not many school transitions
    minimal family disturbances.

What is Reading Comprehension?
  • the process of simultaneously extracting and
    constructing meaning through interaction and
    involvement with written language (RAND, 2002,
    p. 11)
  • Reading is an active and complex process that
  • Understanding written text
  • Developing and interpreting meaning and
  • Using meaning as appropriate to type of text,
    purpose, and situation (NAEP Framework, 2009)

Word recognition, vocabulary, background
knowledge, strategy use, inference-making
abilities, motivation
Text structure, vocabulary, genre discourse,
motivating features, print style and font
Purpose, social relations, school/classroom/peers/
Environment, cultural norms
A heuristic for thinking about reading
comprehension (Sweet Snow, 2003).
Understanding what has been read the
application to written text of (a)
nonlinguistic (conceptual) knowledge (b) general
language comprehension skills
(Rayner, Foorman, Perfetti, Pesetsky,
Seidenberg, 2001)
The Reading Pillar
(NRC, 1998)
Skilled Reading
Speed and ease of reading with comprehension
Conceptual Knowledge/vocabulary Strategic
processing of text
Word Recognition
Decoding using alphabetic principle Decoding
using other cues Sight Recognition
Print Awareness Letter Knowledge Motivation to
Read Oral Language including Phonological
Emergent Reading
What Makes a Text Difficult?
Components of Reading Comprehension (Perfetti,
Comprehension Processes
General Knowledge
Situation Model
Linguistic System Phonology Syntax
Text Representation
Meaning and Form Selection
Lexicon Meaning Morphology Syntax

Word Representation

Orthography Mapping to phonology
Orthographic Units
Phonological Units
Visual Input
Vocabulary Demands in 6 G1 Basals (Foorman et
al., 2004)
(No Transcript)
Some rare (G1 Basal) and not-so-rare
(elementary literature) Words
Representation of Opportunity Words Across Basals
Opportunity Words in Grade 1 Basals
Conclusions on Vocabulary
  • Publishers need to provide teachers with
    cumulative vocabulary lists
  • These need to be made available electronically to
    textbook adopters and should include information
  • Frequency in text and lesson number
  • Separate entry for each definition used
  • Derivational forms
  • Printed word frequency in other relevant corpora

Conclusions on Vocabulary
  • Instruction needs to target oral language
    development from pre-school through high school
  • Printed word frequency and age of acquisition are
    useful tools for guiding selection of lexical
    entries to be taught
  • Assessment of vocabulary for the purpose of
    Reading First should focus on the link between
    assessment and instruction

Summary and Conclusions
  • Programs differ substantially in the composition
    of their print materials for Grade 1 students
  • Length of texts, grammatical complexity, numbers
    of unique and total words, repetition of words,
    coverage of important vocabulary
  • Differences exist in the decodability of types
    and tokens
  • Generally there is greater decodability for
    tokens than types,
  • most programs show improvements for types later
    in the year

Summary and Conclusions
  • Programs vary in the approach they take to
    achieve decodability and in the degree to which
    materials can be expected to yield accuracy in
  • - Vary in phonic elements taught
  • - Vary in opportunity to practice words
    containing these elements
  • - Within 6-week blocks, 70 of words are
    singletons in 4 of the 6 basals
  • - Vary in reliance on holistically-taught words

Implications for fluency
  • for dysfluent readers, the texts that are read
    and reread for fluency practice need to have
    sufficiently high percentages of words withinthe
    word zone fluency curriculum and low percentages
    of rare words, especialy multisyllabic ones ( p.
  • Repetition of core words makes science text
    ideal for fluency practice in the primary grades
    (p. 11)

Hiebert (2007)
Word Zone Fluency Curriculum
High-Freq Words Phonics/Syllable Morphological
A 300 most freq accuracy rate of 40 in first grade in Seymour et al., 2003). Short/long vowels Simple, inflected endings (ed, ing, s, es,s)
B 500 most freq Short long r-controlled vowels
C 1,000 most freq All monosyllabic
D 1,000 most freq 2-syllable compound words with at least 1 root from 1,000 most frequent words Prefixes un, a Suffixes er, est, ly, y (doubling)
E 2,500 most freq
F 5,000 most freq
Jabberwocky (Lewis Carroll, 1872)
Twas brillig, and the slithy toves Did gyre and
gimble in the wabe All mimsy were the
borogoves, And the mome raths outgrabe. Beware
the Jabberwock, my son! The jaws that bite, the
claws that catch! Beware the Jubjub bird, and
shun The frumious Bandersnatch!
And four more stanzas From Through the
Looking-Glass and What Alice Found There
  • You know how to pronounce the words in
    Jabberwocky some are real English words.
  • Which ones are real English words?
  • What is the distinction between those that are
    actual English words and those that arent?
  • Do the two paragraphs differ in these

Alices reaction
  • It seems very pretty, she said when she had
    finished it, but its rather hard to understand!
    (You see she didnt like to confess, even to
    herself, that she couldnt make it out at all.)
    Somehow it seems to fill my head with ideasonly
    I dont exactly know what they are! However,
    somebody killed something thats clear at any

NAEP 2009 Reading Framework
  • Characteristics of text difficulty
  • Vocabulary reported out separately
  • Subscales for literary informational text
  • Grade-level standards for text type

2009 NAEP Framework 2009 NAEP Framework
Literary Text ? Fiction ? Literary Nonfiction ? Poetry Informational Text ? Exposition ? Argumentation and Persuasive Text ? Procedural Text and Documents
Cognitive Targets Distinguished by Text Type Cognitive Targets Distinguished by Text Type
Locate/Recall Integrate/Interpret Critique/Evaluate Locate/Recall Integrate/Interpret Critique/Evaluate
Achievement Levels for Grade 4 NAEP Reading
Achievement Level Literary Informational
Advanced G4 students at the Advanced level should be able to Interpret figurative language Make complex inferences Identify point of view Evaluate character motivation Describe thematic connections across literary texts. G4 students at Advanced level should be able to Make complex inferences Evaluate the coherence of a text Explain authors point of view Compare ideas across texts
Proficient G4 students at the Proficient level should be able to Infer character motivation Interpret mood or tone Explain theme Identify similarities across texts Identify elements of authors crafts G4 students at Proficient level should be able to Identify authors implicitly stated purpose Summarize major ideas Find evidence in support of an argument Distinguish between fact and opinion Draw conclusions
Basic G4 students at the Basic level should be able to Locate textually explicit information, such as plot, setting, and character Make simple inferences Identify supporting details Describe characters motivation Describe the problem Identify mood G4 students at the Basic level should be able to Find the topic sentence or main idea Identify supporting details Identify authors explicitly stated purpose Make simple inferences
2009 NAEP Framework 2009 NAEP Framework 2009 NAEP Framework 2009 NAEP Framework 2009 NAEP Framework
English Mathematics History Science
text type literary informational or technical, symbolic, diagrams expository, argumentative, persuasive Informational or technical, diagrams
text structure plot, setting, characterization, point of view, verse, rhyme sequence, cause and effect, problem and solution, supporting ideas and evidence, graphical features sequence, cause and effect, problem and solution, authors perspective supporting ideas and evidence, contrasting viewpoints, graphical features sequence, cause and effect, problem and solution, supporting ideas and evidence, graphical features
authors craft diction, dialogue, symbolism, imagery, irony, figurative language rhetorical structure, examples, logical arguments figurative language, rhetorical structure, examples, emotional appeal rhetorical structure, examples, logical arguments
(No Transcript)
What Does Mean to be Proficient?
  • W score cutpoints on NAEP and state tests
    communicate grade-level proficiency or benchmark
  • State curriculum standards need to be aligned
    with benchmarks/proficiency levels.
  • Are states proficiency levels comparable to

Proficient on State vs NAEP Reading 2005
State 4-state 4-NAEP DIFF 8-state 4-NAEP DIFF
ME 53 35 -18 44 38 - 6
MO 35 33 - 2 33 31 - 2
WY 47 34 -13 39 36 - 3
TX 79 29 -50 83 26 -57
GA 87 26 -61 83 25 -58
NC 84 29 -55 89 27 -62
Porter, 2007
Most state testing systems do not assess college
and work readiness
  • 26 states require students to pass an exam before
    they graduate high school.
  • Yet most states have testing systems that do not
    measure college and work readiness.

Source Center on Education Policy, State High
School Exit Exams States Try Harder, But Gaps
Persist, August 2005. Source Achieve
Survey/Research, 2006.
Graduation exams in 26 states establish the
performance floor
Figure reads Alaska has a mandatory exit exam in
2005 and is withholding diplomas from students
based on exam performance. Arizona is phasing in
a mandatory exit exam and plans to begin
withholding diplomas based on this exam in 2006.
Connecticut does not have an exit exam, nor is it
scheduled to implement one.
Source Center on Education Policy, based on
information collected from state departments of
education, July 2005.
How challenging are state exit exams?
  • Achieve conducted a study of graduation exams in
    six states to determine how high a bar the tests
    set for students.
  • The results show that these tests tend to measure
    only 8th, 9th or 10th grade content, rather than
    the skills students needs to succeed in college
    and the workplace.

The tests Achieve analyzed
Source Achieve, Inc., Do Graduation Tests
Measure Up? A Closer Look at State High School
Exit Exams, 2004.
Students can pass state English tests with skills
ACT expects of 8th 9th graders
ACT (11th/12th)
ACT PLAN (10th)
ACT EXPLORE (8th/9th)
Source Achieve, Inc., Do Graduation Tests
Measure Up? A Closer Look at State High School
Exit Exams, 2004.
Students Proficient on FCAT(Level 3 and above)
Grade 2001 2006 Difference
3 57 75 18
4 53 66 13
5 52 67 15
6 52 64 12
7 47 62 15
8 43 46 3
9 28 40 12
10 37 32 -5
Is 10th Grade FCAT Too Hard?
  • The St. Petersburg Times article (4/15/07)
    concluded correctly that the 10th Grade FCAT is
    harder than the 10th grade NRT.
  • Conclusion based on fact that Level 3
    (proficient) performance is 56th ile nationally
    at Gr 7 80th ile at Gr 10
  • Or Why wait until high school to implement world
    class standards?

Absolute level of reading proficiency nationally
Grade level standard on the FCAT
Absolute level of reading proficiency
3 4 5 6 7
8 9 10
Passage Length in Words
Grade FCAT range FCAT average NAEP range NAEP average
3 100-700 350
4 100-900 400 200-800
5 200-900 450
6 200-1000 500
7 300-1100 600
8 300-1100 700 400-1000
9 300-1400 800
10 300-1700 900 500-1500 (12) 1000 (12)
of Passage Types
Grade FCAT Literary Texts FCAT Informa-tional Texts NAEP Literary Texts NAEP Informa-tional Texts
3 60 40
4 50 50 50 50
5 50 50
6 50 50
7 40 60
8 40 60 45 55
9 30 70
10 30 70 30 (12) 70 (12)
FCAT Test Design
  • Cognitive Complexity (Webbs Depth of Knowledge)
  • Content Categories for Reading
  • - Words phrases in context
  • - Main idea, plot, authors purpose
  • - Comparison cause/effect
  • - Reference Research locate, organize,
    interpret, synthesize, evaluate information

To Make Proficiency Standards Meaningful and Fair
  • Agree on target for proficiency (e.g., college
  • Align elementary, middle, and high school targets
  • Align curriculum standards
  • Evaluate dimensionality of tests and prepare
    instruction accordingly
  • Equate state tests with NAEP to guarantee
    comparability and equity

From Barbara Tuckmans The Zimmerman Telegram
  • The first message of the morning watch
    plopped out of the pneumatic tube into a wire
    basket with no more premonitory rattle than
    usual. The duty officer at the British Navel
    Intelligence twisted open the cartridge and
    examined the German wireless intercept it
    contained without noting anything of unusual
    significance. When a glance showed him that the
    message was in non-navel code, he sent it in to
    the Political Section in the inner room and
    thought no more about it. The date was January
    17, 1917, past the halfway mark of a war that had
    already ground through thirty months of reckless
    carnage and no gain.

What Makes This Text Difficult?
  • Consider the text type and structure
  • Consider prior knowledge
  • Consider the vocabulary
  • Consider the discourse featureslinguistic
    markers for coherence, coreference, deixis
  • Other factors?

Instructional Considerations
  • Text Type/Structure
  • persuasive text
  • anti-war sentiment, thirty months of reckless
    carnage and no gain
  • indictment of war bureaucracy
  • narrative structure
  • historical non-fiction
  • Prior Knowledge
  • World War I
  • text references war, 1917, British, German, duty
  • early 20th century communications
  • text references telegram, pneumatic tube, wire
    basket, wireless intercept
  • Zimmerman telegram
  • text references German wireless, non-naval code

(No Transcript)
Instructional Considerations (continued)
  • Vocabulary
  • academic language
  • examined, significance, ground through
  • generative words
  • premonitory, carnage, intercept
  • Tier 3 vocabulary (military domain)
  • morning watch, non-naval code, German wireless,
    pneumatic tube
  • Linguistic Markers (Coherence Relations)
  • pronouns
  • duty officer he, him
  • co-references
  • German wireless intercept the message
  • deixis
  • in the inner room
  • chronology
  • When a glance showed him that the message was in
    non-navel code,

Instructional Delivery
  • Model strategies (activating background
    knowledge, questioning, searching for
    information, summarizing, organizing graphically,
    identifying story structure (e.g., Guthrie et
    al., 2004 Brown, Pressley et al., 1996)
  • Keep the focus on the meaning of the text through
    high quality discussion.
  • Model thinking like an historian (e.g.,
    sourcing) to provide a purpose for reading
    (Biancarosa Snow, 2004).

Measuring Text Difficulty
  • Teacher judgment
  • Readability Tuchman passage ranges from 8.4 on
    Dale-Chall to 13.3 on the Flesch-Kincaid Fry
    13.5 on Lexiles.
  • Latent semantic analysis
  • Natural language processing (e.g., McNamara,
  • Text equating to control passage difficulty

Limitations of readability
  • Circular use
  • Capture surface features only
  • Measurement error on specialized text
  • - Primary grade text
  • - Poetry
  • - Technical documents (e.g., train schedules
    tax forms)

How Do We Assess Academic Literacy?
Discussion of Academic Literacy Assessment
  • What are the important knowledge and skills to
    assess in K-3?
  • What are the important knowledge and skills to
    assess in 4-12?
  • What kind of text should be used?
  • What kind of outcome measures should be used?

Converging Evidence
  • Valid and reliable predictors of risk for reading
    difficulty are
  • Print concepts (early K)
  • Letter name knowledge (early K)
  • Phonological awareness and letter sounds (K-1)
  • Rapid naming of letters (end of K to early G1)
  • Word recognition (G1 and beyond)
  • Vocabulary

(Fletcher et al., 2002 Scarsborough, 1998
Torgesen, 2002)
Assessing written language
  • Use various formats to assess
  • --multiple choice
  • --cloze
  • --maze
  • --question/answer
  • --constructed response
  • --retelling
  • --sentence verification
  • Report achievement in language proficiency levels
    to chart ELLs progress (Francis, 2008)

New PK-12 Florida Reading Assessment System
  • Instructionally useful free to FL schools in
  • Includes vocabulary and comprehension
  • Computer administered in grades 3-12
  • Screening, progress monitoring, diagnostic
    assessments data available in the Progress
    Monitoring Reporting Network (PMRN)
  • Screen is empirically linked to the Florida
    Comprehensive Assessment Test (FCAT) or outcome
  • Targeted diagnostic inventories administered to
    students not meeting expectations are linked to
    Florida standards and provide information for
    guiding instruction
  • Reading comprehension oral reading fluency
    passages are equated for difficulty to allow for
    accurate progress monitoring
  • Instructional level passages provided

New Reading Assessments
  • PK print knowledge, phonological awareness,
    vocabulary, math (linked to K screening)
  • K-2 phonemic awareness, letter knowledge,
    decoding, encoding, fluency, vocabulary,
    listening or reading comp.
  • 3-12 adaptive complex low level reading comp.,
    fluency, word analysis, skill assessment
  • K-12 Informal reading inventories
  • Lexile scores in grades 3-12 allow matching
    students to text and access to online libraries
  • Identifies risk of reading difficulties and
    reading disabilities

New Reading Assessments
Thank you!
  • Biancarosa, G., Snow, C.E. (2004). Reading
    nextA vision for action and research in middle
    and high school literacy A report to Carnegie
    Corporation of New York. Washington, DC Alliance
    for Excellent Education.
  • Brown, R., Pressley, M., Van Meter, P.,
    Schuder, T. (1996). A quasi-experimental
    validation of transactional strategies
    instruction with low-achieving second grade
    readers. Journal of Educational Psychology, 88,
  • Foorman, B.R., Francis, D.J., Davidson, K., Harm,
    M., Griffin, J. (2004). Variability in text
    features in six grade 1 basal reading programs.
    Scientific Studies in Reading, 8(2), 167-197.
  • Guthrie, J.T., Wigfield, A., Barbosa, P.,
    Perencevich, K.C., Tabada, A., Davis, M.H.,
    Scafiddi, N.T., Tonks, S. (2004). Increasing
    reading comprehension and engagement through
    Concept-Oriented Reading Instruction. Journal of
    Educational Psychology, 96(3), 403-423.
  • Hiebert, E.H. (2007). A fluency curriculum and
    the texts that support it. In P. Schwanenflugel
    M. Kuhn (Eds.), Creating a literacy curriculum
    Fluency instruction. New York Guilford Press.
  • National Assessment Governing Board (in press).
    2009 NAEP Reading Framework. Washington, D.C.
    Author. Retrieved March 26, 2007 from
  • National Research Council (1998). Preventing
    Reading Difficulties in Young Children. Committee
    on the Prevention of Reading Difficulties in
    Young Children, Commission on Behavioral and
    Social Science and Education. In C.E. Snow, M.S.
    Burns, and P. Griffin (Eds.). Washington, DC
    Natl Academy Press
  • Perfetti, C.A. (1991). Representation and
    awareness in the acquisition of reading
    competence. In L. Rieben C. Perfetti (Eds.),
    Learning to read basic research and its
    implications (pp. 33-44). Hillsdale, NJ Erlbaum.
  • Porter, A. (2007). NCLB lessons learned
    Implications for reauthorization. In A. Gamoran
    (Ed.), Will No Child Left Behind help close
    the poverty gap? Washington, DC Brookings
  • RAND Reading Study Group (2002). Reading for
    understanding Toward a RD program in reading
    comprehension. Arlington, VA RAND.
  • Snow, E., Porche, M., Tabors, P., Harris, S.
    (2007). Is literary enough? Baltimore, MD
  • Snowling, M.J., Hulme, C. (2005). The science
    of reading A handbook. NY Blackwell.
  • Sweet, A.P., Snow, C.E. (2003). Rethinking
    reading comprehension. NY The Guilford Press.
  • Zeno, S.M., Ivens, S.H., Millard, R.T., Duvvuri,
    R. (1995). The educators word frequency guide.
    NY Touchstone Applied Science Associates, Inc.