Computer Mediated Human Tutoring in the Service of Creating an Open Content Tutor: Mass Collaboratio

1
Computer Mediated Human Tutoring in the Service
of Creating an Open Content Tutor Mass
Collaboration for Intelligent Tutoring Systems

• Leena Razzaq Neil Heffernan
• AAAI Fall Symposium
• November 7, 2008

2
Claim

• We will not get the internet classroom unless we
  harness mass collaboration effectively.
• Wiki-pedia and Linux have shown the power of mass
  collaboration
• Nevertheless, Wiki-pedia still makes many folks
  skeptical
• We have a secret weapon we have a reinforcement
  signal - how well kids do on the next question.
• How do we get funding to create the IIC?
• The prospect of Quality research studies
• Around what make for the most learning, for who
  and under what circumstances

3
Outline

• Review what we know about research
• Speculate about how to build the internet
  classroom

4
Randomized Controlled Experiments Investigating
What Works are Important to Funders and the Public

• Mendicino, M., Razzaq, L. Heffernan, N. T. (In
  Press) Comparison of Traditional Homework with
  Computer Supported Homework.  Improving Learning
  from Homework Using Intelligent Tutoring
  Systems.  Journal of Research on Technology in
  Education (JRTE).  Published by the International
  Society For Technology in Education (ISTE).
  Scheduled for the Spring 2009 issue.
• Prawal, Xing, Maharjan, Razzaq, Heffernan
  Heffernan (submitted) The Controversy over
  Tutored Problem Solving Versus Worked Examples
  A Study and Plausible ExplanationProposed.
  Submitted to CHI2009.
• Razzaq, L., Mendicino, M. Heffernan, N. (2008)
  Comparing classroom problem-solving with no
  feedback to web-based homework assistance. In
  Woolf, Aimeur, Nkambou and Lajoie (Eds.)
  Proceeding of the 9th International Conference on
  Intelligent Tutoring Systems. pp. 426 -437.
  Springer-Verlag Berlin.
• Razzaq, L., Heffernan, N. T., Lindeman, R. W.
  (2007). What level of tutor feedback is best? In
  Luckin Koedinger (Eds.) Proceedings of the 13th
  Conference on Artificial Intelligence in
  Education. IOS Press.  pp 222-229.
• Razzaq, L. Heffernan, N.T. (2006). Scaffolding
  vs. hints in the Assistment system. In Ikeda,
  Ashley Chan (Eds.). Proceedings of the Eight
  International Conference on Intelligent Tutoring
  Systems. Springer-Verlag Berlin. pp. 635-644. 
• Croteau, E., Heffernan, N. T. Koedinger, K. R.
  (2004). Why are Algebra word problems difficult?
  Using tutorial log files and the power law of
  learning to select the best fitting cognitive
  model. In J.C. Lester, R.M. Vicari, F. Parguacu
  (Eds.) Proceedings of the 7th International
  Conference on Intelligent Tutoring Systems.
  Berlin Springer-Verlag.  pp. 240-250.
• Heffernan, N. T. Croteau, E. (2004). Web-Based
  Evaluations Showing Differential Learning for
  Tutorial Strategies Employed by the Ms. Lindquist
  Tutor. In James C. Lester, Rosa Maria Vicari,
  Fábio Paraguaçu (Eds.) Proceedings of 7th Annual
  Intelligent Tutoring Systems Conference, Maceio,
  Brazil. Springer Lecture Notes in Computer
  Science.  pp. 491-500
• Heffernan, N. T. (2003). Web-based evaluations
  showing both cognitive and motivational benefits
  of the Ms. Lindquist tutor In F. Verdejo and U.
  Hoppe (Eds)  11th International Conference
  Artificial Intelligence in Education. Sydney,
  Australia. IOS Press.   pp.115-122.
• Heffernan, N. T., Koedinger, K. R.(2002). An
  intelligent tutoring system incorporating a model
  of an experienced human tutor In Stefano A.
  Cerri, Guy Gouardères, Fábio Paraguaçu (Eds.)
  6th International Conference on Intelligent
  Tutoring System. Biarritz, France. Springer
  Lecture Notes in Computer Science  pp. 596-608.
• Heffernan, N. T., Koedinger, K. R. (2000)
  Intelligent tutoring systems are missing the
  tutor Building a more strategic dialog-based
  tutor. In C.P. Rose R. Freedman (Eds.)
  Proceedings of the AAAI Fall Symposium on
  Building Dialogue Systems for Tutorial
  Applications. Menlo Park, CA AAAI Press. ISBN
  978-1-57735-124-5. pp. 14- 19  

5
The ASSISTment System

• A web-based assessment system, designed to
  collect formative assessment data on student math
  skills.
• Students are tutored on items that they get
  incorrect.
• Currently, thousands of students use the system.

6
Comparing Traditional PPH with Homework using
ASSISTments

• Purpose to determine if students can learn more
  by doing their math homework with a web-based
  intelligent tutoring system than when doing
  traditional paper-and-pencil homework.

Mendicino, M., Razzaq, L. Heffernan, N. T. (In
Press) Comparison of Traditional Homework with
Computer Supported Homework.  Improving Learning
from Homework Using Intelligent Tutoring
Systems.  Journal of Research on Technology in
Education (JRTE).  Published by the International
Society For Technology in Education (ISTE).
Scheduled to appear in the Spring 2009 issue.
7
Influences on achievement (Hattie, 1999)
8
Previous work

• Mastering Physics - Warnakulasooriya Pritchard
  (2005) found twice as many students were able to
  complete a set of problems in a given time with
  the help provided compared to students that
  worked on the problems without help.
• Quantum Tutors commercial system claims
  improvement of a full letter grade compared to
  paper-and-pencil homework
• Andes - evaluated in introductory physics classes
  from 1999 2003 (VanLehn et al., 2005) produced
  a mean effect size of 0.6 over paper-and-pencil
  homework.

9
Is WBH practical for K-12?

• Educate teachers on using WBH support
• Free WBH applications (e.g. ASSISTments)
• Narrow the digital divide between students with
  1-to-1 computing programs, new low-cost laptops

10
One to one computing programs

• States such as Maine, Indiana, Michigan and
  Virginia, have begun to implement one-to-one
  computing in schools where each child gets
  his/her own laptop to use during school and often
  to take home.
• The Maine Learning Technology Initiative
  (2002-2004) supplied every Maine 7th and 8th
  grade student and their teachers with laptops,
  with 40 of the middle schools allowing students
  to take their laptops home.
• There are few research studies on the effects of
  one-to-one computing on teaching and learning
• Bonifaz and Zucker, 2004

11
(No Transcript)
12
(No Transcript)
13
(No Transcript)
14
(No Transcript)
15
(No Transcript)
16
(No Transcript)
17
Experiment Design

• 54 5th grade students with internet at home
• 2 problem sets
• Number sense
• Which of the following is closest to the product
  of 397.8 10.3?
• Mixed problems
• 2X 2 14
• What value of X makes the equation shown above
  true?
• Counterbalanced design

18
Experiment design
19
Results

• Students learned significantly more with
  web-based homework assistance with 0.6 effect
  size (p 0.05).

20
Results not just do to a few students
Top 5 gain scores in the web based homework
condition
Our worst gain score is in our WBH condition
21
Implications

• Results of this study are promising, but further
  research is needed to study the impact of WBH
  assistance.
• Could be important to policy makers, particularly
  considering the popularity of one-to-one
  computing initiatives
• Could be more relevant as the digital divide
  narrows and the cost of laptops is dropping

22
What works for whom?

• AIED 2007 experiment
• What type of tutoring for high knowledge versus
  low knowledge students

23
Is more interaction helpful?

• Scaffolding hints represents the most
  interactive experience because students must
  answer scaffolding questions, i.e. learning by
  doing.
• Hints on demand are less interactive because
  students do not have to respond to hints, but
  they can get the same information while solving
  problems as in the scaffolding questions by
  requesting hints.
• Delayed feedback is the least interactive
  condition because students must wait until the
  end of the assignment to get any feedback.

24
(No Transcript)
25
(No Transcript)
26
Delayed Feedback Condition
In this condition, the system behaves the same no
matter what the student answers.
Students get answers and explanations after they
finish all of the problems in the experiment.
27
(No Transcript)
28
Hypothesis

• If the interaction hypothesis is true, students
  in the scaffolding hint condition will learn
  the most. Students in the delayed feedback
  condition will learn the least.
• The effectiveness of the interaction will depend
  on the difficulty of the content.

29
Results

• Conditions were not significantly different at
  pretest.
• Students learned overall from pretest to
  post-test (p 0.005).
• Gain score averages showed a significant
  interaction between condition and math
  proficiency.

30
Gain scores on a single item
All students the interaction between math
proficiency and condition is significant.
31
Results

• Regular students learned more with scaffolding
  hints (p lt 0.05)
• Less proficient students benefit from more
  interaction and coaching through each step to
  solve a problem.
• Honors students learned more with delayed
  feedback (p 0.075)
• More proficient students benefit from seeing
  problems worked out and seeing the big picture.
• Delayed feedback performed better than hints on
  demand for both more and less proficient students
  (p lt 0.05)
• Both more proficient and less proficient students
  do worse when we depend on student initiative.

32
Does an Intelligent Tutor (Ms Lindquist) lead to
more learning than traditional classroom practice?

• Most math classrooms give a lecture and then a
  period of time for practice
• We showed that kids learn a lot more when getting
  immediate feedback from the ITS than the control
  of business as usual
• Immediate feedback is good

• Razzaq, L., Mendicino, M. Heffernan, N. (2008)
  Comparing classroom problem-solving with no
  feedback to web-based homework assistance. In
  Woolf, Aimeur, Nkambou and Lajoie (Eds.)
  Proceeding of the 9th International Conference on
  Intelligent Tutoring Systems. pp. 426 -437.
  Springer-Verlag Berlin.

33
Not only is ITS feedback good for learning, its
also more motivating

• Anderson reports with Cognitive Tutor levels of
  motivation
• Ms. Lindquist study showing students assigned to
  the intelligent tutor conditions will spend a
  long time on the web site.

Heffernan, N. T. (2003). Web-based evaluations
showing both cognitive and motivational benefits
of the Ms. Lindquist tutor In F. Verdejo and U.
Hoppe (Eds)  11th International Conference
Artificial Intelligence in Education. Sydney,
Australia. IOS Press.   pp.115-122.
34

• Transition- And now to speed up the rate of
  talking and hopefully not decrese the
  comprehension rate

35

• Transfer Model
• A list of skills (Ontology in Bev framework)
• A huge list that maps learning object with a
  skill in a given transfer model

36
(No Transcript)
37
(No Transcript)
38
We have shown that our transfer model is good
enough to improve assessment.

• Using finer-grained skill model can improve
  assessment.
• Feng, M., Heffernan, N. T., Mani, M.,
  Heffernan, C. (2007). Assessing students
  performance longitudinally Item difficulty
  parameter vs. skill learning tracking.  The
  National Council on Educational Measurement 2007
  Annual Conference, Chicago.
• Feng , M, Heffernan, N., Heffernan, C. Mani, M.
  (submitted) Using Mixed-Effects Modeling to
  Analyze Different Grain-Sized Skill Models.
  Journal of Technology, Learning, and Assessment.

39
Speculation about how to best get the content of
the IIS built?

• Mass Collaboration
• Teachers, students, parents, researchers
• Example
• Ari Bader-Natals work on SpellBee.com
• Kids play a game where there give each other
  questions and they get points for predicting the
  difficulty of the question
• We could make a game similar to this, but the
  students score goes up if they succeed in
  tutoring their partner as measured by their
  partners ability to get the next problem done
  with no assistance.
• Groups of students that are in a study club that
  get points for tutoring each other, and seeing
  their team proceed.

40
(No Transcript)
41
Tools to Support The Full Life Cycle

• making it easy for teachers to add to
• supporting the full life-cycle for teacher
  creations
• Builder common wrong answer
• Psychometric assistance (running IRT) to tell
  them what items are easy to guess, or have poor
  item functioning (i.e. poor discrimination)

42
Story Telling Time
43

• Stan is a math student in Mr. Kings middle
  school math class and has been designated as a
  qualified student tutor due to the fact that the
  teacher trusts him to help other students and he
  is above average in the class. Stan is working on
  ASSISTments in a classroom where a lot of
  students are at work.
• Across town, at another district school, a
  student in Mr. Smiths class is having trouble
  with Pythagorean Theorem problems. Her name is
  Mary. Mr. Smith and Mr. King know each other from
  district meetings and Mr. Smith trusts Mr. Kings
  choice of student-tutor, so Stan is connected
  electronically to tutor Mary in what, from Marys
  perspective, is a simple chat interface window.
  The computer allows Stan to tutor Mary because it
  knows Stan has already mastered the Pythagorean
  Theorem.
• Stan first asks Mary to state the Pythagorean
  Theorem and then to apply it to this existing
  problem (which side of a triangle corresponds to
  c in the Pythagorean Theorem). This is followed
  by solving the question all the way up to the
  last step involving taking the square root, a
  task that Stan thinks is the hard part. For each
  question he types in the correct answer while
  Mary is thinking so the computer can respond
  instantly instead of always waiting for Stan to
  type. In fact, while Mary is thinking on one
  step, he can proactively write the next question
  or hint message he will give. After a few
  minutes, the tutoring session is over with Mary
  getting the problem solved. Mary is then
  presented, by the computer with a new Pythagorean
  Theorem question. If she gets it correct, it is
  reasonable to infer that the tutoring Stan did
  has some value and that result is used to boost
  the computers belief that Stan created good
  tutoring content.

44

• A few days latter, Stan is online and is asked to
  tutor George on the same question. The computer
  presents to Stan an interface that shows the four
  questions he has already asked in the past, and
  the computer knows the answers already. Stan
  clicks on the first question to again ask the
  student to state the Pythagorean Theorem. Unlike
  with Mary, George does not remember the
  Pythagorean formula so George writes a hint
  message, This is formula that related the length
  of the legs of a right triangle to the length of
  the longest side. George still doesnt get it.
  Stan writes a second hint Its often written
  with a b and c. It turns out that is enough for
  George to say a b c to which Stan responds by
  clicking on the buggy message button and typing
  a message specific to that wrong response abc
  is very close but you need to square all the
  terms
• Those hint messages are saved, as well as the bug
  messages associated with the wrong response, so
  Stan can use them later.
• Two weeks later, Stan has tutored this item 5
  times, and now has a structure of scaffolding
  questions he has refined over time that he likes
  to use. He has a collection of hint messages for
  each question, as well as feedback messages to
  give out for some common answers. This time, Stan
  clicks on the cruise control button and sits
  back and watches his tutoring run all by itself,
  as if he were clicking on buttons to respond. At
  one point Stan notices the student is typing a
  response that he has no buggy message for so he
  turns off cruise control and types in a feedback
  response specific to that error.
• After a few weeks more weeks, Stan has had the
  opportunity to watch his content all on
  cruise-control. In four out of five of those
  cases the student has gotten the later questions
  correct, without needing assistance, which
  provided the computer with evidence that this
  content seems good, so it keeps looking for
  opportunities to present this content, but now in
  cases when Stan is not present.

45
Premise

• Assume 99 of teacher will want to use the IIS
  Certified Content
• 1 will want to use other stuff.

46
How do we scale up what works?

• Most stuff will not work! So its really a
  deletion problem.
• Will involve human editing and judgment or can we
  create a algorithm that will work.
• Multiple level of experimentation

47
Google for educational web page

• We can in effect create a search engine that will
  search the web and rank the quality of all
  educational web pages.
• Can we automate the tagging of skills?

48
How to let people build the transfer model.

• Bev calls this the Ontology Editor. What are the
  knowledge components?
• I assume we need to support communities of
  education researchers as the figure out whose
  ontology to adapt (Rafaels group is building a
  ontology and tagging stuff)
• The hard work is tagging the web with learning
  object knowledge component for a given transfer
  model

49
Research Questions we can answer

• What constitutes good tutoring?
• We can learn that!
• How can we help subject matter experts modify and
  develop ontologies (what are the skills and also
  what ). Some folks will contribute content, some
  will contribute edits in ontologies own by
  groups, some will suggests tagging with a given
  ontology.
• Some will argue about prerequisites hierarchies
  about how to sequence things for a given
  ontology.

50
How groups will own transfer models.

• Since building a transfer model is so hard, we
  need to allow ontology editors to reuse portions
  of other ontologies. New York City, or the State
  of Texas might develop an ontology.
• Quality Control will be related to the ontology
  owners but not the same thing.
• We need to support ontology editors hierarchies
  so that the owner can give worked different
  permissions to adopt things into the hierarchy.
  We to support tools for voting! How a transfer
  model owned by the NCTM executive board can
  revoke editing right of the transfer model.

51
(No Transcript)
52

• Questions?

53

• In how many years will there be something of the
  size and scope of a Wiki-classroom filled with
  good educational content? 2? 5? Surely ten?
• What ideas do you have on how we can leverage
  mass collaboration to create an extensible
  International Internet Classroom?
• How will the set of skills be collaboratively
  developed?
• Who is going to control and maintain such a
  thing?