From Sound to Sense and back again:

1
From Sound to Sense and back again The
integration of lexical and speech processes
David Gow Massachusetts General Hospital
Bob McMurray Dept. of Brain and Cognitive
Sciences University of Rochester
2
The Speech Chain
Complex computations from sound to sense must be
broken up for study.
Assume intermediate representations Phonemes W
ords Syntactic Phrases
3
The Standard Paradigm
Sense
The Standard Paradigm
Words
Phonology
Phonemes
Sound
4
The Standard Paradigm
Sense
The Standard Paradigm
Delimited fields of study.

• Speech Perception

Words

• Spoken Word Recognition

Phonology
Phonemes

• Phonology

Sound
5
Why?
Categorical Perception (CP)
Continuous Acoustic Detail gt Discrete
Categories Does CAD affect speech categorization?
6
Sense
Categorical Perception (CP)

• Defined fundamental computational problems.
• CP is output of
• Speech perception
• Input to
• Phonology
• Word recognition.

Words
Phonology
Phonemes
Sound
7

• But
• Not all speech contrasts are categorical.
• Lots of tasks show non-categorical perception.

Fry, Abramson, Eimas Liberman (1962) Pisoni
Tash (1974) Pisoni Lazarus (1974) Carney,
Widden Viemeister (1977) Hary Massaro (1982)
Pisoni, Aslin, Perey Hennessy (1982) Healy
Repp (1982) Massaro Cohen (1983) Miller
(1997) Samuel (1997)
8
Why has the Standard Paradigm persisted?
Categorical Perception is about phonetic
classification.
The minimal computational problem compute
meaning from sound.
9
Why has the Standard Paradigm persisted?
Even when continuous acoustic detail affects word
recognition, it is seen as outside of core word
recognition.
10
Why has the Standard Paradigm persisted?
Even when continuous acoustic detail affects word
recognition, it is seen as outside of core word
recognition.

• Example Word Segmentation
• Vowel Length
• Stress/Meter
• Coarticulation

Words
Phonemes
Word Recognition
CAD
11
Does continuous acoustic detail affect
interpretation via core word-recognition
processes?

• ? No. Standard Paradigm is fine
• Yes. Hmm

Sublexical Filter (phonemes)

• Need to use stimuli with
• Precise control over CAD

• Need to use tasks that
• reflect only minimal computational problem
  meaning.
• are sensitive to acoustic detail.

12
Visual World Paradigm
Visual World Paradigm

• Subjects hear spoken language and manipulate
  objects in a visual world.
• Visual world includes set of objects with
  interesting linguistic properties (names)
• Eye-movements to each object are monitored
  throughout the task.

Tanenhaus, Spivey-Knowlton, Eberhart Sedivy
(1995) Allopenna, Magnuson Tanenhaus (1998)
13

• Meaning based, natural task Subjects must
  interpret speech to perform task.

• Fixation probability maps onto dynamics of
  lexical activation.

• Context is controlled
• meaning ? lexical activation.

• Eye-movements fast and time-locked to speech.

14
?
Does continuous acoustic detail affect
interpretation? Is lexical activation sensitive
to continuous acoustic detail?
15
McMurray, Tanenhaus Aslin (2003)

• Combine tools of
• speech perception
• 9-step VOT continuum.
• spoken word recognition
• visual world paradigm

16
Methods
A moment to view the items
17
500 ms later
18
Bear
Repeat 1080 times
19
Target Bear Competitor Pear Unrelated Lamp,
Ship
20
VOT0 Response
Time (ms)
21
Predictions
What would lexical sensitivity to CAD look like?
Systematic effect on competitor
dynamics. Fixations to the competitor.
Categorical Results
Gradient Effect
target
target
competitor
competitor
competitor
competitor
22
Results
Response
Response
VOT
VOT
0 ms
5 ms
Competitor Fixations
Time since word onset (ms)
23
Task?
Phoneme ID Not part of minimal computational
problem.
Same stimuli in metalinguistic task more
categorical pattern of fixations
Continuous acoustic detail is not helpful in
metalinguistic tasks
24
Summary
Word recognition shows gradient sensitivity to
continuous acoustic detail.
Not extra-segmental VOT
CAD affects higher-level processes.
Consistent with other studies Andruski,
Blumstein Burton (1994) Marslen-Wilson Warren
(1994) Utman, Blumstein Burton (2000) Dahan,
Magnuson, Tanenhaus Hogan (2001) McMurray,
Clayards, Aslin Tanenhaus (2004) McMurray,
Aslin, Tanenhaus, Spivey Subik (in prep)
25
The Standard Paradigm?
Sense
Words
Phonology
Phonemes
Samuel Pitt (2003) Magnuson, McMurray, Tanehaus
Aslin (2003) Samuel (1997) Elman McClelland
(1988)
Continuous Acoustic Detail
26

The Standard Paradigm?
Sense
From other work
Words
Lexical activation influences sublexical
representations.
Phonology
Phonemes
Phonological regularity affects signal
interpretation.
Continuous Acoustic Detail
Massaro Cohen (1983) Halle, Segui, Frauenfelder
Meunier (1998) Pitt (1998) Dupoux,Kakehi,
Hirose, Pallier Mehler, (1999)
27
Sense
Perhaps interaction and integration make
sense. Do they help solve sticky problems?
?
Words
Phonology
Phonemes
YES
Continuous Acoustic Detail
28

The Emerging Paradigm

• Integration of work in
• spoken word recognition
• speech perception
• phonology

• New computations simplify old problems and solve
  new ones.
• Cognitive processes Lexical activation
  competition.
• Perceptual processes sensitivity to CAD
  perceptual grouping.

29

• CAD is helpful in language comprehension.
• Word segmentation
• Coping with lawful variability due to
  assimilation
• Combination of approaches helps solve both
  problems.

30
Lexical Segmentation
Some lexical processes cant work in the Standard
Paradigm


31
The SWR Solution
? k t I v d I p A ? t m I n t


32
? k t I v d I p A ? t m I n t
active

33
? k t I v d I p A ? t m I n t
active department

34
? k t I v d I p A ? t m I n t
active department
act of dip art mint
a part
depart in

are
par
Standard Paradigm Template matching overgenerates
35
Frauenfelder Peeters (1990)

• Overgeneration resolved through competition in
• TRACE (McClelland Elman 1986)

Problem What if the speaker is trying to say
suck seeds?
36
The Speech Solution

• Cues shown to affect segmentation
• Initial strong syllable
• Initial lengthening
• Increased aspiration
• Increased glottalization

Lehiste, 1960 Garding,1967 Lehiste, 1972
Umeda, 1975 Nakatani Dukes, 1977 Nakatani
Schaffer,1978 Cutler Norris, 1988..
37
Problem cues are subtle and varied,
extra-segmental processes are inelegant
?
Is there a better mechanism?
38
Gow Gordon (1995)
The proposal had a strange syntax that nobody
liked.

The proposal had a strange sin tax that nobody
liked.

• CAD affects interpretation.
• does not trigger segmentation.

39
Good Start Model

• Observation All segmentation cues happen to
  enhance
• word-initial features
• Strengthened cues facilitate activation, making
• intended words stronger competitors

• Incorporating CAD
• Solves overgeneration problem.
• No extra-segmental segmentation process.

Gow Gordon (1995)
40
Summary
When continuous acoustic detail affects lexical
activation, speech and SWR models can be
integrated and simplified


41
Assimilation
The emerging paradigm reframes computational
problems


42
Redefining Computational Problems

• English coronal place assimilation
• /coronal labial/ ? labial labial
• /coronal velar/ ? velar velar

• Standard Paradigm Change is
• discrete
• phonemically neutralizing

43
Standard Paradigm solution Phonological
inference (Gaskell Marslen-Wilson, 1996 1998
2001)
Knowledge driven inference If
labial labial infer /coronal labial/

• greem beans ? green (Gaskell Marslen-Wilson,
  1996 Gow, 2001)

ripe berries ? right (Gaskell
Marslen-Wilson, 2001 Gow, 2002)
Moreover Assimilation effects dissociated from
linguistic knowledge (Gow Im, in press)
44
Assimilation Produces CAD
Assimilatory modification is acoustically
continuous
F2 Transitions in /æC/
Contexts
1850
1800
1750
coronal
Frequency (Hz)
1700
assimilated
labial
1650
1600
1550
Pitch Period
This is not discrete feature change!
45
Regressive Context Effects
Sma
Select the catp box
46
Subject Hears Assim_Non-Coronal (cat/p box)
0.6
0.5
0.4
Fixation Proportion
0.3
0.2
Coronal (cat)
0.1
Non-Coronal (cap)
0
0
400
800
1200
1600
Time (ms)
47
Subject Hears Assim Non-Coronal (cat/p drawing)
0.6
0.5
0.4
Fixation Proportion
0.3
0.2
Coronal (cat)
Non-Coronal (cap)
0.1
0
0
400
800
1200
1600
Time (ms)
48
Progressive Context Effects
Progressive effect in the same experiment
49
Assimilation Use of CAD
Assimilation is resolved through phonological
context.
Partially-assimilated items show regressive
context effects (Gow, 2002 2003) progressive
context effects (Gow, 2001 2003)
Fully assimilated items show neither (Gaskell
Marslen-Wilson, 2001 Gow, 20022003)
50

assimilation context
Infinite regress (eternal ambiguity).
or something more interesting?
51
Continuous acoustic detail is subject to basic
perceptual processes


52
A Perceptual Account
Feature cue parsing (Gow, 2003)
k ? t p b l E d
53
Feature cue parsing (Gow, 2003)
Features encoded by multiple cues that are
integrated
54
Feature cue parsing (Gow, 2003)
55
Feature cue parsing (Gow, 2003)
Assimilation creates cues consistent with
multiple places
56
Feature cue parsing (Gow, 2003)
Extract feature cues
57
Feature cue parsing (Gow, 2003)
Group feature cues by similarity and resolve
ambiguity
58
Feature cue parsing (Gow, 2003)
example eight. catp box
catp drawing catp ?

cor cor
COR cor lab LAB
lab lab
59
Feature cue parsing (Gow, 2003)
example eight. catp Box
catp Drawing catp ?
cor
cor COR cor
lab LAB lab
lab
Progressive and regressive effects fall out of
grouping
60
Summary
SWR problem (eternal ambiguity) replaced by
simpler perceptual problem CAD important in
solution processing obstacle facilitates
perception. Integration of continuous perceptual
features facilitates higher-level
processes. Facilitation via core-word
recognition mechanismsno extra-segmental
routines required.
61
The Standard Paradigm

• Standard paradigm
• Created artificial boundaries that misframed
  issues.
• Continous acoustic detail is variability to be
  conquered..

62
The Emerging Paradigm

• The emerging paradigm
• Emphasis on methodologies that tap the minimal
  computational problem meaning.
• Stresses integration of speech and spoken word
  recognition, questions methods and theory.
• Continuous acoustic detail is useful signal, not
  noise.

63
From Sound to Sense and back again The
integration of lexical and speech processes
David Gow Massachusetts General Hospital
Bob McMurray Dept. of Brain and Cognitive
Sciences University of Rochester