Integrating ICALL into synchronous CMC Can intelligent computer-generated feedback be provided in learner-to-learner interactions?

1
Integrating ICALL into synchronous CMCCan
intelligent computer-generated feedback be
provided in learner-to-learner interactions?

• Markus Dickinson, Soojeong Eom, Natalia Jacobsen,
  Yunkyoung Kang, Chong Min Lee, Ken Petersen,
  Rebecca Sachs
• iicall2 conference December 8, 2007 Waterloo

2
Intelligent CALL (ICALL)

• ICALL provides many potential means of
  facilitating L2 development
• Generation of detailed information regarding
  sources of learner errors
• Precise feedback on learner errors
• Can be tailored to learners common mistakes,
  activity goals, proficiency levels, abilities,
  etc.
• Fosters awareness of relevant language forms
• Tracking of improvement across exercises, using
  learner models
• (cf. Amaral Meurers 2006, Heift Schulze 2007)

3
Intelligent CALL Limitations

• However, there is a tension between
• Contextualized language use is increasingly
  emphasized, and with increasing success (e.g.,
  Amaral et al. 2006 Amaral Meurers 2006)
• But actual communicative interaction is
  relatively unexplored (though see Petersen 2006)
• In order to manage computational complexity
• Exercises are often restricted to the sentence
  level
• Activities often do not simulate true
  communication

The ability of an ICALL system to provide
meaningful, accurate feedback
The flexibility an ICALL system allows for in
terms of meaningful, communicative interaction
4
Synchronous CMC

• Synchronous computer-mediated communication (CMC)
  between L2 learners can provide a beneficial
  context for L2 development
• Focus on meaningful communication
• Activities can approximate target tasks
  relevant to real-life communicative situations,
  with learners functioning as language users
  (Ellis 2003)
• Possibly optimal psycholinguistic environment for
  learners to negotiate for meaning and make
  form-meaning connections (Blake 2000, Doughty
  Long 2003)
• Less pressing time constraints and less ephemeral
  language (e.g., through possibility of
  re-reading) in the written modality
• Reduced processing demands, which may serve as an
  equalizer for learners with lower working memory
  capacity (Payne Whitney 2002)
• Attention to form and monitoring of language
  production (Chapelle 2003, Pellettieri 2000)
• Use of more complex language (Warschauer 1996)

5
Synchronous CMC Limitations

• Concerns regarding the quality of learner-learner
  interactions with respect to potential benefits
  for learning
• The blind leading the blind?
• Without feedback from a trusted authority,
  learners might reinforce each others errors
  (Kern 1995) and/or simply not have the resources
  necessary for correcting each other (Blake 2000)
• In negotiating for meaning, learners may
  naturally tend to focus on lexis without
  attending to morphosyntax (Blake 2000)
• Learners might not be concerned with grammatical
  accuracy (Kern 1995), in which case available
  planning time is a moot point
• ? Teachers may need to set explicit expectations
  for grammatically correct language, while
  simultaneously trying to balance this with a
  primary focus on meaningful communication (Lee
  2001, Pellettieri 2000)

6
Combining the benefits

• Can ICALL and synchronous CMC be integrated in a
  way that exploits the unique benefits of each
  while avoiding their limitations?

detailed, informative, individualized feedback
communicative purpose
negotiation and interactional modifications
attention to form and meaning
approximation of real-world target tasks with
less pressing time constraints
development of functional L2 proficiency with
grammatical accuracy
Intelligent computer-generated feedback in
synchronous, task-based, computer-mediated
learner-learner interactions
7
A balancing act

• We want learners interactions to be as free as
  possible, promoting authentic communication
• At the same time, we are restricted by
  limitations of ICALL processing and must
  constrain the situation somehow (cf. discussion
  in Amaral et al. 2006)
• ? Guided picture-based information-gap tasks can
  constrain the vocabulary and domain in ways that
• reduce many of the complexities involved in
  generating feedback (computationally speaking)
• still allow for communicative interaction between
  learners (interactional authenticity if not
  real-world authenticity)

8
Participants

• English-speaking university students in
  first-year Korean classes
• ?Can (or should) beginning learners use CMC?
• Important for learners to develop the ability to
  use language to communicate meaning as opposed to
  simply displaying language or manipulating L2
  forms systematically without attending to meaning
• However, unconstrained communicative tasks might
  be stressful (or a long shot) for beginning
  learners
• We hope to design our tasks in such a way that
  participants will be comfortable communicating
  meaningfully in the L2
• With appropriate guidance
• Within a constrained and familiar environment

9
A communicative task were considering Spot the
differences

• Each participant sees one version of a house and
  must exchange information in the L2 in order to
  find similarities and differences between the two
  versions
• Not so authentic in terms of real-world
  relevance, but interactionally authentic the
  sort of task often used in interaction research
  to target specific areas of language and promote
  negotiation and L2 learning
• To increase motivation, set up as a competition
  between pairs of participants
• Record (in a provided chart) the activities and
  locations of all characters in partners house
• Indicate whether each of these represents a
  similarity or a difference between the pictures
• Compare scores at the end to those of other
  participant pairs

10
Spot-the-differences task (picture 1)
11
Spot-the-differences task (picture 2)
12
Target of feedback Korean particles

• In Korean (as in Japanese, cf. Nagata 1995),
  postpositional particles are used to indicate
  grammatical functions and thematic roles (e.g.,
  who is doing what to whom).
• They must be used even in simple sentences.
• Particles are taught from the beginning of L2
  Korean study, but
• The system is quite complex and difficult to
  master for adult learners of Korean
• Particle errors account for a substantial
  proportion of the mistakes made by beginning
  learners (Ko et al. 2004)
• Errors persist even at advanced levels

13
Difficulties with Korean particles

• No one-to-one correspondence between Korean
  locative particles and English prepositions
• Korean locative particles mark distinctions not
  made in English

Location of a static object
? (e)
Location of a dynamic activity
?? (eseo)
English in
Location an action moves through
?/? (eul/reul)
(also an object marker)
14
Examples of targetlike and non-targetlike Korean
particle use

• What is in the kitchen?
• ??-? ?-? ???? ??-? ?-? ????
• kitchen-LOC what-SUBJ is kitchen-LOC what-OBJ
  is
• Father is grilling meat.
• ???-? ??-? ???. ???-? ??-? ?? ??.
• father-SUBJ meat-OBJ grill father-SUBJ
  meat-SUBJ is grilling
• A cat is in the living room.
• ???-? ??-? ???. ???-? ??-?? ??.
• cat-SUBJ living room-LOC is cat-SUBJ living
  room-LOC is

(TL examples are on the left with particles in
green non-TL are on the right with particles in
red)
15
ParticipantA ??? ??? ???? ParticipantB ???
???. ParticipantA ??? ???? ????
ParticipantB
CHARACTER LOCATION ACTIVITY
Mother
Father
Grandma ?? ??

S
S
D
D
??? ??? ?? ???.
S
S
D
D
S
D
S
D
S
D
S
D
CHECK
SEND
In your sentence, ?? is marked with the particle
?, which suggests that ?? is an object. Instead,
you need the particle ?? attached to ?? in order
to indicate that ?? is the location of a dynamic
activity.
Word Bank
Particles
?
??
???
?
?
??
?
??
??
?

????
?
????
??
16
ParticipantA ??? ??? ???? ParticipantB ???
???. ParticipantA ??? ???? ????
ParticipantB
TASK PICTURE Partners have slightly different
versions and must communicate to find
differences. They can scroll over the picture to
enlarge it.
CHARACTER LOCATION ACTIVITY
Mother
Father
Grandma ?? ??

S
D
S
D
??? ??? ?? ???.
S
S
D
D
S
D
S
D
CHECK
SEND
S
D
S
D
In your sentence, ?? is marked with the particle
?, which suggests that ?? is an object. Instead,
you need the particle ?? attached to ?? in order
to indicate that ?? is the location of a dynamic
activity.
Word Bank
Particles
?
??
???
?
?
??
?
??
??
?

????
?
????
??
17
ParticipantA ??? ??? ???? ParticipantB ???
???. ParticipantA ??? ???? ????
ParticipantB
CHARACTER LOCATION ACTIVITY
Mother
Father
Grandma ?? ??

S
D
S
D
??? ??? ?? ???.
S
S
D
D
S
D
S
D
CHECK
SEND
S
D
S
D
In your sentence, ?? is marked with the particle
?, which suggests that ?? is an object. Instead,
you need the particle ?? attached to ?? in order
to indicate that ?? is the location of a dynamic
activity.
Word Bank
Particles
WORD PARTICLE BANKS To create a sentence,
participants click on words and particles
?
??
???
?
?
??
?
??
??
?

????
?
????
??
18
ParticipantA ??? ??? ???? ParticipantB ???
???. ParticipantA ??? ???? ????
ParticipantB
CHARACTER LOCATION ACTIVITY
Mother
Father
Grandma ?? ??

which then appear in the sentence drafting area.
S
D
S
D
??? ??? ?? ???.
S
S
D
D
S
D
S
D
CHECK
SEND
S
D
S
D
In your sentence, ?? is marked with the particle
?, which suggests that ?? is an object. Instead,
you need the particle ?? attached to ?? in order
to indicate that ?? is the location of a dynamic
activity.
Word Bank
Particles
?
??
???
?
?
??
?
??
??
?

????
?
????
??
19
ParticipantA ??? ??? ???? ParticipantB ???
???. ParticipantA ??? ???? ????
ParticipantB
CHARACTER LOCATION ACTIVITY
Mother
Father
Grandma ?? ??

If they want help with Korean particle usage,
they can request feedback on their sentences
before entering them into the conversation.
S
D
S
D
??? ??? ?? ???.
S
S
D
D
S
D
S
D
CHECK
SEND
S
D
S
D
In your sentence, ?? is marked with the particle
?, which suggests that ?? is an object. Instead,
you need the particle ?? attached to ?? in order
to indicate that ?? is the location of a dynamic
activity.
Word Bank
Particles
?
??
???
?
?
??
?
??
??
?

????
?
????
??
20
ParticipantA ??? ??? ???? ParticipantB ???
???. ParticipantA ??? ???? ????
ParticipantB
CHARACTER LOCATION ACTIVITY
Mother
Father
Grandma ?? ??

S
D
S
D
??? ??? ?? ???.
S
S
D
D
S
D
S
D
CHECK
SEND
S
D
S
D
In your sentence, ?? is marked with the particle
?, which suggests that ?? is an object. Instead,
you need the particle ?? attached to ?? in order
to indicate that ?? is the location of a dynamic
activity.
Word Bank
Particles
FEEDBACK AREA Here, participants receive
metalinguistic feedback with advice on particle
usage.
?
??
???
?
?
??
?
??
??
?

????
?
????
??
21
ParticipantA ??? ??? ???? ParticipantB ???
???. ParticipantA ??? ???? ????
ParticipantB
CHARACTER LOCATION ACTIVITY
Mother
Father
Grandma ?? ??

S
D
S
D
When they are ready, they click SEND to enter
their utterance into the conversation.
??? ??? ?? ???.
S
S
D
D
S
D
S
D
CHECK
SEND
S
D
S
D
In your sentence, ?? is marked with the particle
?, which suggests that ?? is an object. Instead,
you need the particle ?? attached to ?? in order
to indicate that ?? is the location of a dynamic
activity.
Word Bank
Particles
?
??
???
?
?
??
?
??
??
?

????
?
????
??
22
ParticipantA ??? ??? ???? ParticipantB ???
???. ParticipantA ??? ???? ????
ParticipantB
CHARACTER LOCATION ACTIVITY
Mother
Father
Grandma ?? ??

S
D
S
D
CHAT WINDOW They can scroll up and down to
review the conversation so far.
??? ??? ?? ???.
S
S
D
D
S
D
S
D
CHECK
SEND
S
D
S
D
In your sentence, ?? is marked with the particle
?, which suggests that ?? is an object. Instead,
you need the particle ?? attached to ?? in order
to indicate that ?? is the location of a dynamic
activity.
Word Bank
Particles
?
??
???
?
?
??
?
??
??
?

????
?
????
??
23
ParticipantA ??? ??? ???? ParticipantB ???
???. ParticipantA ??? ???? ????
ParticipantB
GAME RECORD When participants find similarities
or differences, they drag the relevant words for
locations and activities here to record
information about their partners pictures, then
click on S or D to indicate whether the
pictures match in those respects or not.
CHARACTER LOCATION ACTIVITY
Mother
Father
Grandma ?? ??

S
D
S
D
??? ??? ?? ???.
S
S
D
D
S
D
S
D
CHECK
SEND
S
D
S
D
In your sentence, ?? is marked with the particle
?, which suggests that ?? is an object. Instead,
you need the particle ?? attached to ?? in order
to indicate that ?? is the location of a dynamic
activity.
Word Bank
Particles
?
??
???
?
?
??
?
??
??
?

????
?
????
??
24
The importance of instructions

• Piloting so far (with native speakers of Korean
  in face-to-face interactions) has suggested that
  carefully worded instructions will likely be
  needed to
• Encourage participants to use complete sentences
• Emphasize that the characters in the pictures are
  members of the same family (so that terms such as
  mother, brother, etc., can be used naturally)
• Clue participants in to the fact that differences
  will involve the locations and activities of
  family members, as opposed to characteristics of
  furniture (for example)

25
From activity to processing

• How can we process CMC as input?
• ? Built-in constraints
• Intensive feedback is provided on one particular
  error type
• Learners construction of sentences is guided by
• The nature of the picture-based task (constrains
  vocabulary)
• Instructions and the game record chart
• Word and particle banks, which limit the types of
  argument structure by limiting the verbs that can
  be used, and which moreover
• May be necessary for beginning learners who
  cant type in Korean
• May serve as a scaffold for using receptive
  vocab in conversation
• Upshot Processing can focus just on detecting
  particle errors in a known domain.

26
Intensive feedback

• Some have argued that intensive feedback may be
  more effective in certain contexts than
  wide-ranging incidental feedback on a variety of
  error types (e.g., Lyster 1998, Nicholas,
  Lightbown, Spada 2001)
• Intensive feedback focusing consistently on one
  pre-selected error type
• Meta-analyses have not found significant
  differences, but too few primary studies to tell
  for sure in any case, both produce large effect
  sizes (e.g., Mackey Goo 2007, Russell Spada
  2006)
• In our study, we will inform learners that they
  will only be receiving feedback on particles
• Important for meaning (i.e., communicating who is
  doing what to whom) in Korean
• Will hopefully prevent learners from mistaking
  non-feedback for correctness
• Leaves open the possibility of providing other
  feedback, if needed

27
Meaningful communication?

• A spot-the-differences task provides a possible
  forum for learners to use an L2 in purposeful,
  communicative ways, but
• How much will the participants focus on
  meaningful communication if its clear that the
  feedback is focusing exclusively on Korean
  particles?
• ? What else can we do to encourage a balance
  between focusing on intended meanings and
  focusing on the forms required to express those
  meanings accurately?
• How free will the participants be to express a
  creative range of ideas in the context of this
  task?
• ? What can we do to make the word bank
  sufficiently rich for their communicative
  purposes?
• How fluid will communication feel if participants
  have to take the time to search for each word
  they want to use?
• ? What can we do to make the words in the bank as
  easy to access as possible?

28
Future directions

• Pilot the game with L2 learners
• Get a clearer sense of what to expect in learner
  input
• Test how the word bank actually plays out
• Develop the system
• Activity, error diagnosis, and feedback modules
• Processing techniques
• Make the tasks more complex, meaningful, and
  relevant to real-life communicative situations
• Create realistic rationales for having to find
  differences (e.g., a detective and witness
  discussing a crime scene as it appears just
  before and after a crime)
• Explore other sorts of picture-based
  information-gap tasks
• Remove or expand the word bank
• This would require additional constraints,
  however, to keep processing feasible
• Develop activities to target more constructions

29
Opportunity to test techniques

• A combination of techniques will ultimately be
  used.
• Linguistic processing will be kept separate from
  error diagnosis and feedback generation.
• We can use this setting to experiment with
  different processing technologies
• More traditional, anticipation-based regular
  expressions
• Basic linguistic abstraction of POS tags and
  chunks to provide annotation of input (cf. Amaral
  Meurers 2006)
• Experimental ill-formed input checking
• Use mismatches in 2 different parsers to detect
  particle errors (cf. Metcalf Boyd 2006)
• One parser captures particle usage patterns from
  real language
• One parser captures general argument structure
  patterns between words, irrespective of particles

30
Opportunity to test questions of SLA theory and
language pedagogy

• We will be able to explore learners uses of
  various types of feedback in CMC, using pre-test
  / treatment / post-test designs to assess L2
  development
• Intensive vs. non-intensive feedback?
• Metalinguistic information vs. prompts vs.
  recasts? (etc.)
• Content-oriented feedback?
• Eventually, we can use such a set-up to
  investigate ways of optimizing feedback
• For different areas of language
• For learners of different proficiency levels
• For learners with different aptitude profiles
• Using and integrating this system with the Korean
  language curriculum at Georgetown

31
Questions or comments?

• Please email

Markus Dickinson md7_at_indiana.edu
Soojeong Eom se48_at_georgetown.edu
Natalia Jacobsen ndj5_at_georgetown.edu
Yunkyoung Kang yk95_at_georgetown.edu
Chong Min Lee cml54_at_georgetown.edu
Ken Petersen kap22_at_georgetown.edu
Rebecca Sachs rrs8_at_georgetown.edu
32
References

• Amaral, L., Meurers, D. (2006). Where does
  ICALL fit into foreign language teaching. CALICO
  2006. University of Hawaii.
• Amaral, L., Metcalf , V., Meurers, D. (2006).
  Language awareness through re-use of NLP
  technology. Pre-conference Workshop on NLP in
  CALL Computational and Linguistic Challenges.
  CALICO 2006. University of Hawaii.
• Blake, R. (2000). Computer-mediated
  communication A window on L2 Spanish
  interlanguage. Language Learning Technology,
  4(1), 120-136.
• Chapelle, C. (2003). English language learning
  and technology. Philadelphia John Benjamins.
• Doughty, C.J., Long, M.H. (2003). Optimal
  psycholinguistic environments for distance
  foreign language learning. Language Learning
  Technology, 7(3), 50-80.
• Ellis, R. (2003). Task-based language learning
  and teaching. Oxford Oxford University Press.
• Heift, T., Schulze, M. (2007). Errors and
  intelligence in computer-assisted language
  learning Parsers and pedagogues. Routledge.

33
References

• Kern, R. G. (1995). Restructuring classroom
  interaction with networked computers Effects on
  quantity and characteristics of language
  production. The Modern Language Journal, 79,
  457-473.
• Ko, S., Kim, M., Kim, J., Seo, S., Chung, H.,
  Han, S. (2004). An analysis of Korean learner
  corpora and errors. Hanguk Publishing Co.
• Lee, L. (2001). Online interaction Negotiation
  of meaning and strategies used among learners of
  Spanish. ReCALL Journal, 13(2), 232-244.
• Lyster, R. (1998). Form in immersion classroom
  discourse In or out of focus? Canadian Journal
  of Applied Linguistics, 1, 53-82.
• Mackey, A., Goo, J. (2007). Interaction
  research in SLA A meta-analysis and research
  synthesis. In A. Mackey (Ed.), Conversational
  interaction and second language acquisition A
  series of empirical studies (pp. 407-452).
  Oxford Oxford University Press.
• Metcalf, V., Boyd, A. (2006, December).
  Head-lexicalized PCFGs for verb subcategorization
  error diagnosis in ICALL. Workshop on Interfaces
  of Intelligent Computer-Assisted Language
  Learning. The Ohio State University.

34
References

• Nagata, N. (1995). An effective application of
  natural language processing in second language
  instruction. CALICO Journal, 13(1), 47-67.
• Nicholas, H., Lightbown, P. M., Spada, N.
  (2001). Recasts as feedback to language
  learners. Language Learning, 51, 719-758.
• Payne, J. S., Whitney, P. J. (2002). Developing
  L2 oral proficiency through synchronous CMC
  Output, working memory, and interlanguage
  development. CALICO Journal, 20(1), 7-32.
• Pellettieri, J. (2000). Negotiation in
  cyberspace The role of chatting in the
  development of grammatical competence in the
  virtual foreign language classroom. In M.
  Warschauer R. Kern (eds.), Network-based
  language teaching Concepts and practice, 59-86.
  Cambridge CUP.
• Petersen, K. (2006, December). Measuring L2
  development in an ICALL context. Workshop on
  Interfaces of Intelligent Computer-Assisted
  Language Learning. The Ohio State University.
• Russell, J., Spada, N. (2006). The
  effectiveness of corrective feedback for the
  acquisition of L2 grammar. In J. M. Norris L.
  Ortega (Eds.), Synthesizing research on language
  learning and teaching. Amsterdam John Benjamins.
• Warschauer, M. (1996). Comparing face-to-face and
  electronic discussion in the second language
  classroom. CALICO Journal, 13(2), 7-26.