Multimodal reformulation during shared synchronous note-taking : potential pedagogical consequences for teachers and students Kristine Lund*, Gregory Dyke

1
Multimodal reformulation during shared
synchronous note-taking potential pedagogical
consequences for teachers and studentsKristine
Lund, Gregory Dyke, Jean-Jacques
GirardotICAR Research Lab, CNRS, University of
LyonRIM Department, Ecole des Mines de St.
Etienne
2
Overview

• Motivation
• Note-taking and multimodal reformulation
• The corpus
• A series of tool-supported meetings
• tutoring computer programming projects at lEcole
  des Mines de St. Etienne
• Tatiana (Trace Analysis Tool for Interaction
  ANAlysts)
• Synchronization of multi-source data, analyses
  and visualizations
• Results
• Form and content / pedagogical consequences ?
• Synthesis and perspectives

3
Note-taking

• Listen, understand and assimilate, choose what to
  record, study (Suritsky Hughes, 1991)
• Modify definition for our context(e.g. also
  reading, but no trace of studying)
• Look for patterns in note-taking and determine
  efficiency in relation to students results
• The value of detailed notes rises with time
  between the time notes are taken and the test
  (Williams Eggert, 2002)
• A strong correlation between the quantity of
  notes taken and exam performance (Nye, et. al.
  1984).
• Note-taking as a tool for transforming knowledge
  and not as data collection (Castello Monereo,
  2005)
• We focus on the process of note-taking

4
Multimodal interactions (and reformulations)

• Modes
• Abstract resources non-material that help us to
  make sense
• touch, hearing, smell, taste (senses),
• gesture, speech (means of expression)
• Media
• Specific material forms in which modes are
  expressed
• The human body, a mechanical body, computer
  (physical apparatus)
• Example of a mode
• Manipulate an object to write
• Example of four media

Kress Van Leeuwen (2001) Bellik Teil (1992)
5
Reformulation

• Collaborative writing / note-taking context
• An oralo-graphic situation, articulates two
  different semantic modes
• Apotheloz (2001) Cooperation in this task
  consists in showing in a continuous way, each
  step of the way, the manner in which what is
  formulated articulates with what has already been
  formulated

6
A multimodal corpus (talk, writing)

• First year computer programming projects at Ecole
  des Mines in St. Etienne
• 33 tutoring sessions, lasting from 30 to 60
  minutes a piece (PPF --gt 12 transcribed)
• 9 groups
• 3 project types, each chosen by three groups.
• These 9 groups were tutored by 2 professors.
• Computer traces of note-taking and transcribed
  dialogue
• The computer program / final product
• Final reports
• The presentations
• Grades

7
Exploratory research questions

• What are the nature of the multimodal
  reformulations ?
• How can they demonstrate conceptual
  transformations ?

?
8
Tatiana - Trace Analysis Tool for Interaction
ANAlysts

• Synchronize multi-source data
• Computer traces of human interaction
• Videos and their corresponding transcription
  (dialogue, etc.)
• Replay in order to understand activity
• Define analysis units and categorize facets of
  the interaction
• Visualize the categorizations on a temporal line
• Regroup, re-categorize, re-visualize
• Share

9
Synchronize the multi-source data, replay and
think about units of analysis / Research Q
Transcription of talk
Video
Shared text editor where notes are taken
(replayed)
Log file
10
Create the note-taking writing units (done
manually)
Log file
Note-taking units
11
Categorize the reformulated - reformulation pair
(done manually)
Categorize the transcription of talk
Categorization schema
Categorize the note-taking units
12
Create the links between the reformulated -
reformulation pair (manual)
Categorized transcription of talk
Categorized note-taking units
Directional link between the reformulated and the
reformulation
13
Visualization (automatically created)
Transcription of talk
As note-taking
Bs note-taking
The reformulated
The reformulation

• Synchronize multi-source data
• Replay in order to understand activity
• Define analysis units and categorize facets of
  the interaction
• Visualize the categorizations on a temporal line
• Regroup, re-categorize, re-visualize
• Share

14
Tatiana - example analysis of corpus
Tool replayer (5)
Log Data (1)
Writing units (2)
Video (6)
Transcription (3)
Transcription
Writing units
Visualization (4)
Remote control (7)
Reformulation
Highlights are synchronized
15
Results - session 1
16
A writing conflict provokes a spatial organization
Results - session 2
17
Chronological time-lag between monologue and
note-taking
Results - session 2
18
Differences in the structure of reformulation
Results - session 2
19
Reformulation of tutor dialogue written and
co-elaborated by two students
Results - session 2
20
Tutor reaction upon seeing note-taking
Results - session 2
21
Synthesis, interpretations and questions

• Multimodal reformulation, co-elaborative written
  reformulation of dialogue
• Not yet seen oral reformulation of co-elaborative
  note-taking
• From written to oral
• Response to a student question
• Incite students to take note-taking further
  (conceptual deepening)
• Is this still reformulation ?
• From oral to written
• Anticipation of teacher talk do such students
  understand content better ?
• How to understand the choices students make when
  reformulating ?
• What is the linguistic nature of these
  reformulations ?
• How do the characteristics of the shared text
  editor influence note-taking and co-elaboration?
  (technical affordances of resources)
• Collaborative note-taking ?
• For what type of content is there a time-lag
  between teacher talk and note-taking ?

22
Perspectives

• More closely study the nature of these multimodal
  reformulations
• How can they illustrate knowledge transformation
  ?
• We dont look to measure student performance in
  terms of exam results as a result of note-taking.
• Rather, we want to quality the transformation of
  conceptual knowledge through characterizing
  reformulation.
• What are the pedagogical consequences ?

NO

• How can the shared text editor characteristics
  influence note-taking ?
• More or less co-elaboration ?
• Can note-taking be understood as a sort of
  formative (vs. summative) evaluation ?
• What about auto and hetero-regulation ? And
  vicarious learning?

23
Thanks to the PPF !

• Transcription of dialogue from videos
• Communication between TATIANA and CARTE
• Collection, activity Analysis and Regulation
  based on Traces Enriched (Christophe Courtin,
  Stéphane Talbot / Chambery)
• to facilitate automating certain analysis tasks
  (e.g. definition of note-taking units,
  categorization)
• to highlight relevant sequences of events in
  large numbers of TATIANA traces
• CARTE supplies the following software tools
• convert TATIANA traces CARTE traces and vice
  versa
• import/export CARTE traces to and from a corpus
  of traces
• Analyze CARTE traces by means of rules
  (pre-defined patterns)

24
You can download Tatiana

• http//code.google.com/p/tatiana/
• gregory.dyke_at_emse.fr
• kristine.lund_at_univ-lyon2.fr
• jean-jacques.girardot_at_emse.fr
• steven.collins_at_emse.fr