Evaluating the Effectiveness of Textual Feedbacks and Predictions in Algorithm Visualizations

1
Evaluating the Effectiveness of Textual Feedbacks
and Predictions in Algorithm Visualizations

• By
• Mehmet Celal Dasiyici, Haeyong Chung, Sung Hee
  Park, and Puranjoy Bhattacharjee
• celalchunghshparkpuran_at_vt.edu

2
Outline

• Introduction
• Engagement
• Motivation
• The Goal
• Experimental Design
• Versions of the AV
• Independent Dependent Variables
• Procedure
• Questionnaires
• Participant Recruitment Methods
• What to Do Next / Project Plan

3
The effectiveness of AV

• The effectiveness of AV
• It can be determined by its pedagogical values.
• Pedagogical values
• How much students learn about that algorithm by
  using the AV?
• How can we increase the pedagogical benefits of
  using the AV?
• Not what users see, but how they engage and
  interact with the visualization increases the
  effectiveness of an AV.

4
Engagement

• Engagement with AV can increase the pedagogical
  value of the AV (Hundhausen et al. 2002)
• Lack of significant improvement is often
  attributable to the AV not actively engaging the
  learners
• Not what the users passively see, but how they
  engage and interact with the visualization
• Construct their own input dataset, make
  prediction, program the algorithm, and answer
  questions about the algorithm being visualized,
  construct their own visualization

5
Motivation

• Prediction Make prediction about the next
  algorithm steps
• Our study is motivated by two studies Byrne el
  al. 1996 and Jarc et al 2000. Both conducted
  experiments to evaluate the benefit of
  prediction
• They failed to show significant impact of
  predicting the next steps of the algorithm on the
  learning outcomes.
• Hypothesis being able to predict the next steps
  of the algorithm would improve learning an
  algorithm when it is coupled with the textual
  feedback about each answer the user gives.

6
The goal

• Evaluate pedagogical effects of textual feedback
  which is coupled with prediction process in the
  AV
• Define better ways of designing more effective
  AVs
• Increase pedagogical benefits of the usage of an
  AV.

7
Experimental Design

• Between-subjects design crossing different
  versions of an AV viewing an AV.
• Independent Variables Versions of the AV
• (a) No textual feedback/no prediction,
• (b) Textual feedback/no prediction.
• (c) Textual feedback/prediction.
• Dependent Variables
• Post-test scores Pre-test scores

8
Experimental Design

• Procedure
• Pre-test
• Examine one version of the visualization.
• Post-test
• Population Virginia Tech students or Blacksburg
  residents. (N10 for each version, N30 in total)
• Prediction Users will have more performance
  improvement when the level of engagement between
  user and AV increases.
• Algorithm Visualizations To Be Used We will use
  hashing tutorials quadratic probing section in
  our experiments.

9
Questionnaires

• Pre-test and post-test will be taken by
  participant before and after treatment,
  respectively.
• Pre-test and post-test will be identical
  questions.
• 10 questions in each test two type of questions
• 4 easy questions (A question 1-4)
• 6 difficult questions(B question5-10)
• Some questions ask how to use a basic hash
  function, in this evaluation, mod function(3, 6,
  7).
• Some questions ask how to use a collision
  resolution method, in this evaluation, quadratic
  probing (the rest of them)
• 12 1,
• 22 2, 4, 5, 8
• 32 9
• 42 10

10
Participant recruiting methods

• Thirty students will participate in the study,
  all volunteers and all undergraduate and graduate
  students in computer science at Virginia Tech.
• They had had little or no exposure to Hashing -
  Quadratic probing topic, but all had taken either
  undergraduate- or graduate-level algorithms
  courses.
• All records are private. Researchers will not use
  any personal information, including name, email,
  sex, age, etc.
• Recruiting poster, department mailing lists, and
  course mailing list will be used.
• Recruiting poster and department mailing lists
  will be used.

11
what to do/ plan