M

1
Mémoire, Apprentissage et Langage Aspects
Cognitifs

• Peter Ford DOMINEY
• Institut des Sciences Cognitives, CNRS UMR 5015,
  UCB Lyon 1

2
(No Transcript)
3
Plan

• Langage Constructions grammaticales
• Structure des séquences sensorimotrices
• Réseau Neuronal
•  Equivalence  apprentissage langage et
  séquences abstraites
• Bases neurophysiologiques
• Ré-éducation
• Apprentissage multi-langue
• Apprentissage robotique
• Fin

4
Constructions Grammaticales et mots de fonction
The dog gave the elephant to the monkey.
Gave (Agent, Object, Recipient)
The elephant was given to the monkey by the dog.

• Same meaning different syntactic structure
• Different transformation from sentence to meaning
• Signaled by function words (the, was, to, by)
• Function words trigger transformation/mapping
  rules to associate open class nouns to their
  thematic roles (agent, object, recipient).

5
Structure Abstraite et Constructions
Grammaticales
Construction Di-transitif

• Constructions Grammaticales
• Mapping ltForme sensgt
• Langage
• Ensemble structuré de constructions grammaticales
• Correspondance entre
• Constructions grammaticales
• Structures abstraites
• Hypothèse dÉquivalence

Structure Abstraite  Di-Transitif 
J D B M D J B M
Dominey Evolution of Communication 02, Dominey et
al., Brain and Language 03
6
Structure des Séquences Sensorimotrices
A B C

• Sérielle (ordre)
• ABCBACB
• CABACBA
• Temporelle (rythme)
• A BCAB AC A
• AB C A BAC A
• Abstraite (règle)
• ABC-BAC
• DEF-EDF
• .
• 123 213

Prefrontal Working Memory
Input
WM
A B C
Recog
Cortex
State
Selection
Recurrent Connections
Output
Striatum
Reward-related dopamine synaptic plasticity
BAC
Réseau Neuronal

• Dominey et al. JCogNeuro 95, Dominey
  BioCybernetics 95
• Dominey Boussaoud CogBrRes 97
• Dominey JEP 98, Dominey CogBrainRes 98
• Dominey BrainLang 97, Dominey et al. JCogNeuro
  98, Dominey Ramus Lang Cog Proc. 2000,
  Dominey et al. 1997, Lelekov et al. 2000

7
Model learns to perform thematic role assignment

• Training
• Present sentence one word at a time
• Indicate agent, object and recipient
• Learn transformational relations so State can
  Modulate appropriate STM to Output
• Testing Correction
• Present new sentences
• Model emits the correct roles
• Error-Learning
• 100 Correct Caplan

The elephant was given to the monkey by the dog.
Open Class Stream (Content)
Input
Closed Class Stream (Function) the, was, to, by
STM
elephant monkey dog
Recog
State
Selection
Output
Canonically ordered nouns agent(dog),
object(elephant), recipient(monkey)
Dominey, et al, Brain and Language, 2003.
8
Theoretical Implications

• Functional relation between linguistic and
  non-linguistic abstract sequence processing
• Common neurophysiological basis for rule-based
  structural transformation processing

9
Corrélats Neuro-Linguistiqueschez des patients
aphasiques agrammatiques

1. Perturbations Corrélées entre Séquences
   Abstraites - Langage
2. Transfert dapprentissage entre Séquences
   Abstraites - Langage

1. Perturbations Corrélées
2. Transfert dapprentissage
Lelekov, Dominey et al. 00 Dominey et al.
BrainLanguage 03, Hoen et al. NeuroReport 03
10
Corrélats Neuro-Linguistiqueschez des patients
aphasiques agrammatiques
11
Neurophysiology - ERP Shared cortical networks
for abstract sequence and language

• ERP
• Common Cortical Network
• Langage/Abstract Seq
• LAN, P600, N400
• ABCXBAC vs ABCYABC

Compare Sentence and Sequence Processing for
closed class elements Sentence The faucet
was soldered by the plumber Sequence ABC
X BAC
(Michel Hoen - Thesis 2003)
12
Corrélats Neurophysiologiquesen IRM fonctionnelle

• Réseau cortical commun pour le traitement des
  transformations structurelles
• Réseau cortical distinct pour lintégration de la
  semantique dans ces transformations

Newman 2003
IRMf
Langage
Langage Seq
Hoen, Dominey et al. Cortex 2005 special issue
on Integrative models of Brocas area and the
ventral premotor cortex
13
Modèle Actualisé Traitement des Constructions
Grammaticales
Langage Séquence (Pars opercularis) Ba
44 Transformations Structurelles
Langage spécifique (Pars triangularis) Ba
45 Intégrations Sémantiques
IRMf
Langage
Séquences
ConstructionIndex
ABC X BAC
__ __ __ X __ __ __
Construction Inventory
A B C
B A C

Dominey Hoen Cortex 2005 Special Issue on
Integrative Models of Brocas Area
14
Grammar and Grammatical Constructions Learned

• The block pushed the triangle.
• The block gave the triangle to the moon.
• The triangle was pushed by the block.
• The moon was given to the triangle by the block.
• The block that pushed the triangle touched the
  moon.
• The block pushed the triangle that touched the
  moon.
• The block pushed the triangle that was touched by
  the moon.
• The block that was pushed by the triangle touched
  the moon.
• 35 syntactic structures

• 1. S NP VP
• 2. NP ? Det N
• 3. NP ? NPRel VP
• 4. VP ? Va NP
• 5. VP ? Va NP PP
• 6. VP ? Aux Vp PP
• 7. VP ? Aux Vp PP PP
• 8. PP ? Prep NP
• 9. Det ? the, a
• 10. N ? cylinder, block, moon
• 11.Va ? touched, pushed, took, gave
• 12.Vp ? touched,pushed, taken, given
• 13. Aux ? was
• 14. Prep ? to, by, from
• 15. Rel ? that

15
Cross-Linguistic Validation Japanese

• Free word order
• Grammatical roles of NPs indicated by
  post-positions
• Will the cue competition model adapt to this
  configuration?
• -ga, -ni, -yotte, -to and wo markers as closed
  class elements

16
Cross-Linguistic Validation Japanese

• Block-ga circle-wo oshita triangle-ni-yotte
  tatakareta.
• Circle-wo oshita triangle-ni-yotte block-ga
  tatakareta.
• The block was hit by the triangle that pushed the
  circle.
• Pushed(triangle, circle), Hit(triangle, block)
•  
• Circle-wo tataita block-ga triangle-ni-yotte
  osareta.
• Triangle-ni-yotte circle-wo tataita block-ga
  osareta.
• The block that hit the circle was pushed by the
  triangle.
• Hit(block, circle), Pushed(triangle, block)
•  
• Block-ga circle-wo oshita triangle-wo tataita.
• Circle-wo oshita triangle-wo block-ga tataita.
• The block hit the triangle that pushed the
  circle.
• Pushed(triangle, circle), Hit(block, triangle)

N 26 constructions
17
Using These Learned Grammatical Constructions
for Event Description and Interrogation
Vision, "Meaning" Extraction
CCD Camera
Event(Agent, Object,
Recipient)
Spoken Language Interface (CSLU RAD)
Human narrator
Sentence
E(A,O,R)
Grammatical Construction Model Sentence to
Meaning
Gave(moon, cylinder, block)
The moon gave the cylinder to the block. The
block was gave the cylinder by the moon. The
cylinder was gave to the block by the moon.
18
Grammatical Constructions for Robot Command

• Grammatical constructions
• Link between command meaning (predicate-argument
  structure) and sentence structure
• Sentence - Meaning mappings
• Event description
• Action(Agent, object, recipient)
• Sentence
• Commanding
• Sentence
• Move(Object, location)

19
Plan

• Langage Constructions grammaticales
• Structure des séquences sensorimotrices
• Réseau Neuronal
•  Equivalence  apprentissage langage et
  séquences abstraites
• Bases neurophysiologiques
• Ré-éducation
• Apprentissage multi-langue
• Apprentissage robotique
• Fin