Mining Data from Randomized Within-Subject Experiments in an Automated Reading Tutor

1
Mining Data from Randomized Within-Subject
Experiments in an Automated Reading Tutor Joseph
E. Beck and Jack Mostow Project LISTEN
(www.cs.cmu.edu/listen), Carnegie Mellon
University Funded by National Science Foundation
and The Heinz Endowments
Experiments embedded in the Reading Tutor help
evaluate its decisions in tutoring decoding,
vocabulary, and comprehension
Research question What tutorial decision does
the experiment investigate? Trial context In
what situation does the tutorial decision
occur? Randomized decision The Reading Tutor
chooses at random among plausible alternative
actions. Each such choice starts an experimental
trial. Randomizing the decision allows causal
attribution. Trial outcome We define the
outcome of each decision based on subsequent
student behavior a much finer-grained and more
copious source of data than post-test
scores. Analysis Aggregating over many such
trials can tell not only which choices work best,
but when and for whom.
Decoding What type of help is most effective
for helping students learn to decode
words? Student is reading story and clicks on
a word for help Examine student
performance on a future encounter of the word.
Does the student ask for help? Does the tutor
accept the word as read correctly? N
189,039 help events Rhyming help is most
effective overall. For hard words, best to just
tell the student the word.
Vocabulary Does a brief introduction to a
words meaning before a story help the student to
learn the word and comprehend the story? Before
student starts to read story, Reading Tutor
identifies vocabulary words in story.
While student reads story, assess
comprehension of story. After
student finishes story, assess retention of
vocabulary. N 5,668 vocabulary words
tested Explaining words helps for both
within-story comprehension probes and after-story
vocabulary questions but effects interact with
reading level.
Comprehension Does inserting generic wh-
questions help students comprehend
stories? Student is reading story
N15,187 cloze questions Logistic regression
model
Reading Tutor randomly picks half of vocabulary
words to explain
Reading Tutor randomly selects which type of help
to provide
Reading Tutor randomly decides whether to insert
a wh- question

Draw
Rhymes with saw
Student continues reading story
During the story, student encounters cloze
questions
Two outcome measures
Independent variables Helps/Hurts p
preceding wh- questions ? 0.023
preceding cloze questions
recent wh- questions ? 0.074
recent cloze questions
Time since prior question (sec) ? 0.036

This work was supported by the National Science
Foundation under ITR/IERI Grant No. REC-0326153.
Any opinions, findings, and conclusions or
recommendations expressed in this publication are
those of the authors and do not necessarily
reflect the views of the National Science
Foundation or the official policies, either
expressed or implied, of the sponsors or of the
United States Government.