Adapted typographical annotations for language learning: userprofiled design solutions to problems o

1
Adapted typographical annotations for language
learning user-profiled design solutions to
problems of auditory perception and speech
production
Anthony Stenton, Lairdil, Université
Toulouse I A3H_at_AH06
anthony.stenton_at_univ-tlse1.fr
2
CNRS Project 2003-2005 TCANTraitement Cognitif
Apprentissage et NTIC

• The development of a prototype authoring system
  using visual cues for improving the perception
  of stress patterns in foreign languages, in
  particular in English
• SWANS Synchronised Web Authoring Notation
  System

3
Fields of study of SWANS project specialists

• Linguistics
• Auditory Perception
• Cognitive Psychology
• Computer Science
• Document design specialists

4
Laboratories involved

• LAboratoire Interuniversitaire de Recherche en
  DIdactique des Langues, (LAIRDIL) Toulouse,
• Anthony Stenton, Anne Péchou, Gail Taillefer,
  Nicole Décuré, Antoine Toma, Christine Vaillant
  Sirdey

5

• Laboratoire Travail et CognitionUMR CNRS 5551,
  Toulouse
• André Tricot
• Laboratoire de Neuropsycholinguistique
  Jacques-Lordat, EA 1941 Toulouse
• Pascal Gaillard, Michel Billières, Angelika
  Rieussec

6

• Laboratoire d'Analyse et d'Architecture des
  Systèmes, LAAS CNRS 7, Toulouse
• Saïd Tazi
• Equipe Automatique des Procédés
  Biotechnologiques, Laboratoire de
  Biotechnologies-Bioprocédés, INSA Toulouse
• Nabil Kabbaj

7
Project objectives

• Measurable progress in stress perception, and
  oral production
• The development of an authoring system for
  synchronising and annotating texts for an
  adaptive multilingual context
• Allowing teachers to annotate without having to
  master advanced programming skills

8
The problem of French student oral presentations
in English at advanced levels

• Fluent
• Students find their words easily
• Appropriate use of language
• But.

9
Random stress patterns

• Most francophone speakers do not consciously
  store information on stress patterns. During
  oral presentations , stress is placed randomly on
  the second or third syllable of the word
   deVELopment  indicating not deafness or a
  problem of production but negligence of memory
  storage.
• Communication depends on tolerance thresholds.
  It can breakdown after only a few stress errors.

10
The challenges

• To define the rules of English stress (Alain
  Deschamps 94, daprès Guierre) and an appropriate
  learning method (Théo van Leeuwen Visual
  Design, Rudolph Arnheim Visual Weight)
• To develop protocols for evaluating learning
  (Sweller, Mayer, Paivio)
• To develop an Intelligent Environment for Human
  Learning where visual perception can help
  improve auditory perception
• To develop a prototype authoring system to make
  the process of textual annotation semi-automatic

11
70 to 90 of our French students get the accent
wrong

• /"gVv._at_n.m_at_nt/ government
• /"st.Is.faI/ satisfy
• /"In.t_at_r.est/ interest
• /kO.p_at_r"eI.S_at_n/ corporation
• /"kO.p_at_r._at_t/ corporate
• / _at_"nl._at_.sIs/ analysis
• /"sen._at_t/ senate
• /"prez.I.d_at_nt.si/ presidency
• /"mIl.I.tri/ military
• //"p._at_.Ti/ apathy

12

• Why ?

13
Mother-tongue interference

• French stress is on the right
• (a codachrone language)
• English stress is on the left
• (a capochrone language)
• Reading and subvocalizing ( silent reading) can
  reinforce bad habits

14
A psychological barrier

• In terms of pronunciation, one might argue, the
  Hundred Years War is far from ended. Many French
  students refuse to leave the comfortable
  foothills of their own relatively low pitch
  frequencies for a roller coaster ride in English
  high frequencies. These are French or Latin words
  on the screen. Why on earth should I employ such
  unnatural stress patterns?

15
Are the problems with English stress patterns the
same for all nationalities?

• No
• In Czech, Latvian, Hungarian, Swiss German
  (Bernese dialect), Finnish, and Swahili, stress
  is always placed on the first syllable.
• In French, Spanish, Portuguese, Turkish, and
  Polish stress is rarely placed on the first
  syllable.

16
Italian compound nouns

• Stress usally on the second element
• Transfer to English loan words leads to
•  airBAG  instead of  AIRbag 

17
Textual annotation must adapt to student
mother-tongues

• Language centres must offer texts which place
  annotations on words where English the stress
  pattern will probably cause problems of
  perception for novice learners according to
  their mother-tongue

18
A battering ram of a solution

• It is suggested that adult L2 learners might
  counter L1 interference effects through managing
  to receive exaggerated acoustic cues, multiples
  instances of many talkers, and massed listening
  experience (McCandliss et al 2002 McClelland et
  al.1999)

19
Hypothesis of dual coding (Paivio)

• Dual coding (sound text paralinguistic
  markers) should improve learning for novice
  learners while having no effect or a negative
  effect for those who already know the place of
  the accent.

20
Textual annotation techniques for learning
pronunciation

• How far can we go ?
• When does the brain start to complain because
  reading becomes too slow or because codes seem
  too obscure ?
• What constitutes cognitive overload ?

21
Robertson method circa 1850
22
The Robertson Method
23
Brazil 1994
24
Sounds Right 2002
25
Can typography change ?

• Is our software capable of offering new more
  memorable techniques for learning pronunciation ?
  
• Are we ready for a more flexible, animated and
  colourful alphabet ?

26
Annotation techniques
27
A text annotated with Swans
28
Question Time in Parliament annotated with SWANS
29
A Spanish text
30
German
31
Cimabue 1300 Rogier
van der Weyden 1450
32
Does reading on a computer screen change learning
?
33
Simultaneous reading and listening

• Cinema 1931 subtitles to the Jazz Singer in
  Danish
• BBC television 1938, opera subtitles
• Karaoke 1970s
• Language Learning Audio Partner Teleste 1995

34
How does memory behave after dual coding ?
35
Should synchronised reading be intensive or
extensive ?
36
Does synchronised reading inhibit L1
interference during silent reading?
37
How does memory behave when we change colours?
38
How does memory behave when text moves ?

• Three  levels of representation for words in
  memory  ( triple storage)
• conceptual level
• semantic-syntaxical level (lemmes)
• formal level (lexèmes)
• Willem Levelt, Modèle de la production langagière
  
• (résumé from de Bot, Paribakht Wesche 1997,
  312)

39
Levelts analysis is based on student reading of
paper documents

• Toulouse tests of dual coding at the word level
  and at the sentence level constitute an early and
  limited analysis of the potential of the new
  plasticity in computer based reading
• Over 200 students participated
• Sound recordings of student pronunciation were
  stored to examine stress perception

40
Tests with students in Toulouse
41
Problem The adult brain is conditioned by
typography

•  Freeing ourselves from chirographic and
  typographic bias is probably more difficult than
  any of us can imagine  Ong 1982

42
Annotation techniques
43
Synchronisation with SMIL

• The arrival of SMIL (Synchronised Multimedia
  Integration Language) in 1999 and the work of the
  W3 Consortium - Timed-Text Initiative (2003)
  - constitute an important breakthrough. SMIL
  allows fine-tuned synchronisation of sound and
  text combined with a heightened degree of
  typographical plasticity for on screen textual
  annotation.

44
How do you synchronise sound and text ?
45
Problem the absence of annotated and
synchronised texts

• Programming by hand with Magpie and
  Dreamweaver, it takes two hours to annotate
  and synchronise a short text with a film or a
  sound file and to transfer the code into a web
  page.
• Early testing required 50 documents in 3
  languages English, Spanish German

46
Solution 2005 SWANS

• Synchronised Web Authoring Notation System
• The programme generates SMIL code which is placed
  automatically in web pages to allow students to
  study stress by reading and listening
• The generation of annotated and synchronised web
  pages has been cut from 2 hours per one page
  document to 10 minutes.

47
The four stages of authoring with SWANS
48
SWANS Synchronised Web Authoring Notation System
49
Annotation with SWANS
50
Can annotation become automatic?

• Like spelling correctors accuracy levels are
  variable but acceptable
• Method 1 rule-based expert system
• According to Deschamps we can deduce the stress
  pattern of a word from its spelling in 90 of
  cases. Some rules, such as those concerning
  endings with or the rule,
  are close to a 100 (Deschamps p.33, 1994) and
  are relatively easy to implement. Accordingly all
  such character chains occurring in a given text
  can be temporarily highlighted in SWANS to aid
  annotation. Unstressed suffixes can be
  highlighted in red (e.g.  -able, -age, -ance,
  -ancy, ant, -cy, -ful, -hood, -ist, -ise, -ize,
  -less, ly, ment, -ness, -or/-er, -ous) while
  stressed suffixes (-ade, -aire, -ee) are
  highlighted in blue if the user desires.

51
Can annotation be automatic ?

• Method 2 voice recognition techniques
• The integration of voice recognition
  techniques giving raw and optimised phonemic
  transcriptions adapted from the PERL algorithms
  of the Aix-Marsec project is a more complex
  challenge. The Aix-MARSEC project time-aligns
  speech data at the phoneme and syllable level.
  Aix- MARSEC tools consist of a set of reference
  files (grapheme-phoneme conversion dictionaries)
  and multiplatform Praat and Perl scripts.

52
Can annotation become automatic?

• Method 3 text banks of annotated texts
• a bank of pre-annotated scripts, from 2004-2005
  3-minute video news bulletins, offers an
  alternative to rule-based or dictionary based
  systems (e.g. 71,000 entries of the Oxford
  Advanced Learners using the SAMPA alphabet)
  but, of course, requires manual adjustment for
  ambiguous cases (convict the verb
  /k_at_n"vIkt/ or the noun /"kQn.vIkt/ ?) .

53
Conclusions

• Synchronised reading based on dual coding
  techniques opens promising new perspectives for
  working on pronunciation in an adaptive
  multilingual context.
• Mother-tongue based annotation techniques offer
  an adaptive solution to individual learning needs
  which deserves further exploration in
  university language centres.

54
The extensive reading hypothesis

• Extensive reading of synchronised text can help
  students to inhibit interference from the L1
• Perception of stress improves in novice learners
  exposed to sound and annotated text.
• Improved oral production cannot be demonstrated
  in the short term and may take time to appear.
  Explicit knowledge of stress patterns is a new
  springboard for turning savoir into savoir faire
  and savoir être.

55
Does SWANS announce the Swan-Song of the
Gutenberg tradition for language learners
studying pronunciation ?

• Resistance to changing reading habits may ensure
  an erratic evolution of reform but we believe the
  way is open for new experiments with intensive
  and extensive reading on the computer screen with
  potentially important repercussions for
  perception and oral production.

56
Will teachers use SWANS ?

• The eternal debate should teachers become
  authors of on-line resources ?
• Testing will take place in several European
  language centres affiliated to CERCLES in 2006