Merging the senses

1
Merging the senses

• Course 3.3B
• Perception, Imagery, and Brain
• Department of Psychology
• Maastricht University
• Bernadette Schmitt

2
Merging of the senses

• Vision and Audition
• Why does watching a speakers lips improve speech
  perception? (Calvert et al., 1997)
• Why is there an interaction at al?
• (McGurk effect)

3
McGurk effect
4
Two stations (at least)

• Superior temporal polysensory system (STP)
• Superior colliculus

5
Kosslyns model of vision

• Input into visual buffer (retina - occipital
  lobe)
• parallel processing of what and where
  (dorsal, ventral)
• convergence of what and where in associative
  memory (STP)
• information lookup (dorsolateral prefrontal,
  posterior parietal)
• attention (superior colliculus)

6
Audition

• Analysis of sound into frequency components
• cochlea basilar membrane, haircells
• parallel processing of time and intensity
• Two parts of cochlear nucleus
• Magnocellular nucleus (time)
• Nucleus angularis (intensity)
• convergence of time and intensity
• inferior colliculus
• higher order audition
• Primary auditory cortex (frequency)
• Secondary auditiory cortex (speech sounds,
  melodies)

7
From sound to vibration

• Sound wave creates vibration of the conductive
  apparatus

8
The signal travels along the auditory nerve
9
The destination is the auditory cortex
6
10
Where do V and A meet?

• one hint in Kosslyn
• associative memory
• visual, auditory, and sensory information
  converge
• STP (superior temporal polysensory system)

11
STP location

• Superior temporal
• lobe

12
STP location

• Superior temporal
• sulcus

13
STP location

• Superior temporal
• sulcus
• dorsal bank and
• fundus

14
STP investigated

• Bruce, Desimone,
• and Gross (1981, 1986)
• single cell recording
• in Macaque

15
Properties of the STP neurons

• all responded to visual stimuli
• large receptive fields
• sensitive to movement
• sensitive to faces
• insensitive to form, size, or contrast
• 50 of the neurons also response to other senses
• 21 to visual and auditory stimuli
• 17 to visual and somesthetic stimuli (touch)
• 17 were trimodal

16
The modalityof neurons
17
Where does the activation come from?

• Afferent basis of STP (Input from)
• inferior temporal cortex (whose neurons are
  visual)
• superior temporal cortex (whose neurons are
  auditory)
• posterior parietal cortex (touch)
• Suggestion I
• STP merges the senses

18
Where does the activation come from?

• Afferent basis of STP (vision, audition, touch)
• Input from other polysensory areas
• lateral and orbital frontal cortex (association
  cortex)
• cingulate gyrus(attention move window)
• parahippocampal gyrus (part of limbic associaiton
  cortex)
• Suggestion II
• Merging of the senses happens elsewhere

19
Where does the activation come from?

• Afferent basis of STP
• Input from from other polysensory areas
• Input from medial pulvinar (Thalamic nuclei)
• Thalamus processes visual and auditory
  information
• pulvinar gets input from deeper layers of the
  superior colliculus

20
Superior Colliculus location

• Midbrain
• Close to
• inferior
• colliculus

21
Superior Colliculus outfit

• Several layers
• superfical
• deep

22
Superior Colliculus (SC) investigated

• Wallace, Wilkinson, and Stein (1996)
• single cell
• recording
• in Macaque

23
Properties of the SC neurons

• Neurons at superficial layer responded to visual
  stimuli
• Neurons showed retinotopic order

24

• Retinotopic
• mapping

25
Properties of the SC neurons

• Neurons at superficial layer responded to visual
  stimuli
• Neurons showed retinotopic order
• sensitive to specific movement directions
• sensitive to specific velocity
• sensitive to stimulus onset or offset

26
Properties of the SC neurons

• Neurons at superficial layer responded to visual
  stimuli
• Neurons at deep layers
• sensitive to vision, acoustic stimuli, and touch
• multisensory sensitive to combination of
  different stimuli modalities
• receptive fields of different modalities match in
  space

27
Properties of the SC neurons

• The visual field goes with the auditory space

28
Properties of the SC neurons

• The visual field goes with the auditory space
  goes with boddy areas

29
Properties of the SC neurons

• The visual field goes with the auditory space
  goes with boddy areas

30
Multisensory performance

• Multisensory neurons integrate cues from
  different modalities
• WHAT HAPPENS TOGETHER GOES TOGETHER
• together in time
• together in space

31
Multisensory performance (time)

• What happens at the same time goes together
• The neuron has to take into account differences
  in temporal processing
• Visual information comes in late auditory
  information comes in early
• 100 ms to visual stimuli
• 10-20 ms to acoustic stimuli and touch
• the amount of sensory interaction in a neuron
  therefore depends on the onset of the stimuli

32
Multisensory performance (time)

• Time course
• of visual and
• acoustic
• interaction
• in a neuron

Visual first Acoustic second
Acoustic first
33
Multisensory performance (space)

• What happens at the same place goes together
• For example
• an object appears in front of the monkey
• a sound comes from the front of the monkey
• The neuron that is sensitive to the front fires
• But it fires in a very interesting way

34
Multisensory enhancement
35
What is the consequence ofmultisensory
enhancement?

• not sure
• but maybe in behavior
• multisensory enhancement in superior colliculus
  might be one explanation for crossmodal
  facilitation in perception.
• It might be one answer to our initial question
• Why is it easier to understand someone if one can
  watch the speakers lips?