Improving Comprehension Online Project, 2005-08

1
Improving Comprehension Online Project, 2005-08
Designing and testing a universally designed
strategic digital literacy environment for
diverse learners Bridget Dalton, Vanderbilt
University Patrick Proctor, Boston College IES
Research Conference Washington, DC June 11,
2008
A goal 2 development award to CAST, Inc.
2
Research team

• Bridget Dalton (Co-PI), Vanderbilt University,
  and Elaine Mo, Kristin Robinson, Ge Vue, Mary
  OMalley, Boris Goldowski, CAST, Inc.
• Patrick Proctor (Co-PI), Yi-Chien Li, Kevin
  OConnor, Boston College
• Catherine Snow (Co-PI), Paola Uccelli, Sabina
  Neugebauer, Lorena Landeo Schenone, Harvard
  Graduate School of Education
• School partners 3 semi-urban and 1 urban school
  in northeastern Massachusetts

3
Project goal

• To develop and test a universally designed
• (Rose Meyer, 2002) strategic digital
  reading approach (Dalton Proctor, 2007) to
  improving reading achievement of 5th grade
  students, including bilingual students and
  struggling readers

4
Multiple perspectives required
5
Universal design for learning (Rose Meyer, 2002)

• Design for the broadest range of learners from
  the beginning avoid retrofitting
• Provide multiple means of
• Representation
• Expression
• Engagement

6
Assumptions

• Shift to universal design perspective
• Attention to diversity and individual difference
  benefits individual and society
• New literacies, while more complex, are more
  flexible and inclusive
• potential to level the playing field for those
  who have not fared well with print literacy

7
Rand Reading Study Groups (2002) reading
comprehension heuristic
reader
reader
text
text
activity
activity
comprehension
comprehension
Sociocultural context
Sociocultural context
8
Strategic Digital Reading (Dalton Proctor,
2007)
reader
text
activity
comprehension
Sociocultural context
9
Comprehension in a new literacies landscape
Strategic digital reading
reader
text
activity
comprehension
Sociocultural context
10
How does ICON support diverse learners in
relation to

• Representation?
• Expression?
• Engagement?
• What is unique for ELLs?
• What features/supports are essential for some
  good for many/all?

11
Iterative design, formative feedback and testing
Y2. Compare Vocabulary, Strategies Combo
Y2. Compare Vocabulary, Strategies Combo
Versions
Y1. Develop Vocabulary
Y1. Develop Vocabulary
Y3. Compare Combo Vs. Control
Y3. Compare Combo Vs. Control
12
ICON optimal prototype (Yr. 3)
13
Embedded Strategies
14
Spanish language support
15
Coaches
Level 1 coaches provide text-specific models and
think alouds. As skill increases, students
select strategies and coaches provide generic
think-alouds.
16
Vocabulary Connect It!
17
Vocabulary Language Alert
More than 60 of the power words are Spanish
English cognates
18
Vocabulary Web It!
19
Vocabulary Caption It!
20
All 3 years/studies Feasibility, appeal
usability

• Teachers and students view ICON as a helpful
  reading tool, easy to use, engaging
• Technical support required bandwidth issue
• Variation in teacher enactment of ICON suggests
  need for additional study
• English proficiency levels influence ways in
  which students use ICON and extent to which
  additional support is needed
• Peer collaboration one means of support
• Increased sensitivity to learner (needs, use of
  system, performance) is likely to benefit all

21
Y1 Study of Semantic Depth

• 35 students, 24 bilinguals (Spanish and other
  low-incidence languages), 11 English monolinguals
• Oral language skills (WJ picture vocablistening
  comprehension)
• Reading skills (WJ passage comp MCAS ELA score)
• Average semantic depth score for 8 target words
  (Anxiously, Bitter, Dense, Grasp, Ignore,
  Menacing, Powerless, Relieved)

22
Yr. 1 study of semantic depth(Proctor, Uccelli,
Dalton, Snow, in press)

• Effective teaching and learning activities
  targeted
• for further analysis Caption-It

23
Semantic depth was a significant predictor of
reading performance, mediated by interaction with
English oral language proficiency
No effect for language status (bilingual vs.
monolingual)
24
What did we learn from Y1 vocabulary study?

• Caption It Encouraging but preliminary As both
  an activity and an assessment, it appeared to
  reduce the monolingual/bilingual gap in students
  performance, though oral language was heavily
  implicated
• Semantic Depth Promising but far from final
• Semantic depth showed a positive association with
  reading comprehension, beyond the contribution of
  decoding and oral language skills
• Semantic depth seems to play a more prominent
  role as oral language skills improve
• Semantic depth refers to a cluster of skills
  associations among these and with other
  dimensions of vocabulary depth need to be
  explored.

25
Year 2 study Strategies vs. vocabulary vs. combo

• Very hard to find research that compares effects
  of vocabulary versus comprehension instruction
• Likely because the two are so strongly
  intertwined
• Given that vocabulary is a primary focus of
  instruction for ELL students, we found this
  question intriguing, and asked 2 basic questions
• Does assignment to condition (Vocabulary-only,
  Strategy-only, Combo) affect students
  performance on standardized and
  researcher-developed measures of vocabulary and
  comprehension?
• Do the effects vary by language status
  (monolingual, Spanish-English bilingual, other
  bilingual)?

26
Our hypotheses

• For standardized and researcher-developed
• vocabulary
• Combo gt Vocabulary gt Strategy
• For standardized and researcher-developed
• comprehension
• Combo gt Strategy gt Vocabulary

27
Y2 study of vocab vs strategies What matters
and for whom?(Dalton, Proctor, Uccelli, Mo,
Snow, in preparation)

• 106 students, 21 Spanish-English bilinguals, 17
  other-English bilinguals, 68 monolinguals in 6
  classrooms, 3 districts
• Random assignment to condition (vocabulary,
  strategy, combination)
• 14-week intervention
• Pre-post standardized testing, embedded
  vocabulary and comprehension quizzes

28
Y2 effect size overview
29
What did we learn from Y2 study of vocabulary vs.
reading strategies vs. combined?

• Overall, hypotheses held, and combination version
  showed strongest results across standardized and
  researcher measures
• Did the small sample size mask learner by
  treatment interactions?
• Theoretically, an interaction between student
  characteristics (reader type and/or language
  status) would make sense
• Sample size may be too small
• Student controls access to support and may not be
  making good decisions about when and how to use
  support
• Thresholds of language proficiency

30
Year 3 study Combo vs control

• Having established general effectiveness, time to
  move to
• a comparison between treatment and control using
  optimal
• version of ICON
• Quasi-experimental study
• 12 classrooms, classrooms assigned randomly to
  treatment or control condition, n 227 (108
  control, 119 intervention 10.5 other
  bilinguals, 48.5 Spanish-English bilinguals, 41
  English monolinguals)
• For intervention group, 2 x 50 minutes per week,
  for 16 weeks
• For control group, across the three districts,
  standard literacy curriculum included reading
  strategies focus, but limited vocabulary
  instruction
• Initial training of teachers and students by
  research team, gradual release of ICON prototype
  teaching responsibility
• Analyses conducted at student level,
  randomization at teacher level

31
Y3 measures

• Gates-MacGinitie reading vocabulary and
  comprehension subtests pre- and post-intervention
• Aprenda reading vocabulary - pre
• Researcher-designed breadth of vocabulary
  (targeted words) post-intervention
• 20-item multiple choice assessment
• Reseacher-designed depth of vocabulary,
• post-intervention
• 5-item definition, drawing, captioning
  assessment

32
Y3 results

• General results
• No effect of condition on standardized measures
  significant voc and comp gain for both groups
• Strong effect of condition on researcher
  developed measures

Condition
Depth/Breadth
Standard Vocab and comp
33
Condition effects on ICON vocabulary breadth
Significant effect of condition on ICON voc.
Breadth F(1,205) 56.62, p lt .001 Significant
difference between Spanish bilinguals and English
monolinguals (t 5.1, p lt .001) Strong readers
significantly outperform average (t 5.1, p lt
.001) and struggling (t 12.6, p lt .001) No
interactions by language or reader status and
condition
34
Condition effects on ICON vocabulary depth
Exp. significantly outperform Control on
vocabulary depth F(1,224) 101.4, p lt
.001 English monolinguals significantly
outperform Spanish bilinguals (t 5.3, p lt .001)
and non-Spanish bilinguals (t 2.2, p lt
.05) Strong readers significantly outperform
average (t 5.0, p lt .001) and struggling (t
9.4, p lt .001) readers No interactions by
language status, BUT average-reader X condition
interaction (p .048)
35
Spanish-English bilinguals and ICON depth of
vocabulary
For intervention Spanish-English bilinguals,
Spanish Vocabulary scores explained English
Vocabulary Depth performance, after controlling
for condition and prior English proficiency.
36
Conclusions and next steps

• Continue to analyze Year 3 data
• Worklog responses, multimodal retellings, student
  feature use, teacher use of feedback support.
• For whom does this intervention work best?
• Goal 3 effects for Spanish-English bilinguals
  and struggling readers are intriguing
• Goal 2 Work for transfer. Design for increased
  sensitivity to learner characteristics,
  especially language proficiency.
• Distal effects on standardized measures