Refined Micro-analysis of Fluency Gains in a Reading Tutor that Listens

1
Refined Micro-analysis of Fluency Gains in a
Reading Tutor that Listens

• Jack Mostow and Joseph Beck
• Project LISTEN (www.cs.cmu.edu/listen)
• Carnegie Mellon University
• Consultant and Scientific Advisory Board Chair,
  Soliloquy Learning
• Society for the Scientific Study of Reading
• 13th Annual Meeting, July, 2006
• Funding National Science Foundation, Heinz
  Endowments

2
Research questions and approach

• Guided oral reading builds fluency NRP 00
• Typically repeated oral reading
• but how do its benefits vary?
• How good is repeated vs. wide reading?
• How good is massed vs. spaced practice?
• How do the answers vary with student proficiency?
• Approach micro-analyze oral reading data
• Massive hundreds of children
• Longitudinal entire school year
• Fine-grained word by word

3
Project LISTENs Reading Tutor Rich source of
guided oral reading data

• Massive
• 650 students age 5-14
• Mostly grades 1-4
• Longitudinal
• 2003-2004 school year
• 55,000 sessions
• Fine-grained
• 6.9 million words
• Heard by recognizer

• Video at www.cs.cmu.edu/listen

4
Reading speeds up with practice example

• Initial encounter of muttered
• Ill have to mop up all this (5630 ms) muttered
  Dennis to himself but how
• 5 weeks later (different word pair in different
  sentence)
• Dennis (110 ms) muttered oh I forgot to ask him
  for the money
• Word reading time latency production time ?
  1/fluency
• How does word reading time change in general?

5
Learning curve for mean reading time of first 20
encounters, excluding top 50 words

• Do some types of encounters help more than others?

6
Four types of word encounters
New context? First time today?
1. Read muttered in a new story. Wide Spaced
2. Read muttered in another sentence. Wide Massed
3. On a later day, reread sentence 1. Reread Spaced
4. Then reread sentence 2. Reread Massed

• Predict reading time for 770,858 type 1
  encounters
• from prior encounters of all 4 types.

7
Predictor variables

• Number of word encounters so far of each type
• Wide vs. reread
• Spaced vs. massed
• Word difficulty
• of letters
• of past help requests (controls for difficulty
  for that student)
• Student proficiency
• WRMT Word Identification grade-equivalent score,
  e.g. 2.3
• Interpolated for each encounter from pre- and
  post-test scores

8
Exponential model of word reading time

• L letters (P proficiency constant A)
  e - learning rate B Exposure
• Define weights for each type of encounter
• r for rereading vs. 1 for wide reading
• m for massed vs. 1 for spaced
• h for help requests
• Exposure weighted sum of of word encounters
  so far
• 1 of wide, spaced encounters
• r of reread, spaced encounters
• m of wide, massed encounters
• r m of reread, massed encounters
• h of help requests
• Beck, J. Using learning decomposition to analyze
  student fluency development. ITS2006 Educational
  Data Mining Workshop, Taiwan.

9
Analysis

• Use SPSS non-linear regression to fit parameters
• Caveat 770,858 trials are not independent
• So be conservative
• Split 650 students into 10 groups
• Fit r, m, for each group
• From the 10 estimates of each parameter, compute
• Mean standard error
• Differs significantly from 1?

10
Overall results

• Wide reading beats rereading
• r .68 .13
• r lt 1 (p .007)
• 2 new stories 3 old stories
• Spaced beats massed practice
• m .67 .13
• m lt 1 (p .007)
• 2 spaced encounters 3 massed encounters
• Do these results vary by proficiency?

11
Effects of proficiency
Bottom third Middle third Top third
Word ID GE 1.8 (0-2.3) 2.7 (2.3-3.1) 4.5 (3.1-10.2)
Reread (r) .93 .23 .99 .23 .79 .25
Massed (m) 1.73 .46 .41 .08 .41 .21

• When does wide reading beat rereading?
• Maybe only for high readers?
• Seeing a word again the same day
• May help low readers more than waiting (p .058)
• Helps higher readers less than seeing it later (p
  lt .01)

12
Conclusion type of practice matters!

• Wide reading beats rereading
• At least for higher readers
• Advantage of spaced practice varies with
  proficiency
• Low readers seeing a word again the same day
  may help more
• Higher readers better to wait
• Fluency growth is slow (learning curve is
  gradual)
• So differences in practice quality are hard to
  detect
• But possible by micro-analysis of massive,
  longitudinal, fine-grained data
• Future work
• Clarify interaction with proficiency
• Refine model of fluency practice
• Test correlational results experimentally
• Thank you! Questions?
• See papers videos at www.cs.cmu.edu/listen

13
Predictive models of word reading in text
SSSR2005 SSSR2006
Predict Growth from word encounter i to i1 Performance at encounter i1
Outcome Reading time speedup Reading time, errors, help requests
Predictor Encounter i of word Encounters 1..i
Model Linear Exponential
14
Outcome variable

• Combine reading time, errors, help requests
• Cap reading time at 3 seconds (0.1 of data)
• Treat error as 3 seconds
• Treat help request as 3 seconds