Active Vision

1
Active Vision
Carol Colby Rebecca Berman Cathy Dunn Chris
Genovese Laura Heiser Eli Merriam Kae
Nakamura Department of Neuroscience Center for
the Neural Basis of Cognition University of
Pittsburgh Department of Statistics Carnegie
Mellon University
2
Why does the world stay still when we move
our eyes?
Hermann von Helmholtz Treatise on Physiological
Optics, 1866
Effort of will
3
(No Transcript)
4

• Remapping in monkey area LIP and extrastriate
  visual cortex

5

• Remapping in monkey area LIP and extrastriate
  visual cortex
• 2) Remapping in split-brain monkeys
• Behavior
• Physiology

6

• Remapping in monkey area LIP and extrastriate
  visual cortex
• 2) Remapping in split-brain monkeys
• Behavior
• Physiology
• 3) Remapping in human cortex
• Parietal cortex
• Striate and extrastriate visual cortex

7
LIP memory guided saccade
Stimulus On
Saccade
8
Stimulus appears outside of RF
Saccade moves RF to stimulus location
9
Single step task
10
Spatial updating or remapping The brain combines
visual and corollary discharge signals to create
a representation of space that takes our eye
movements into account
11
LIP Summary Area LIP neurons encode attended
spatial locations.
12

• LIP Summary
• Area LIP neurons encode attended spatial
  locations.
• The spatial representation of an attended
  location is remapped when the eyes move.

13

• LIP Summary
• Area LIP neurons encode attended spatial
  locations.
• The spatial representation of an attended
  location is remapped when the eyes move.
• Remapping is initiated by a corollary discharge
  of the eye movement command.

14

• LIP Summary
• Area LIP neurons encode attended spatial
  locations.
• The spatial representation of an attended
  location is remapped when the eyes move.
• Remapping is initiated by a corollary discharge
  of the eye movement command.
• Remapping produces a representation that is
  oculocentric a location is represented in the
  coordinates of the movement needed to acquire the
  location.

15

• LIP Summary
• Area LIP neurons encode attended spatial
  locations.
• The spatial representation of an attended
  location is remapped when the eyes move.
• Remapping is initiated by a corollary discharge
  of the eye movement command.
• Remapping produces a representation that is
  oculocentric a location is represented in the
  coordinates of the movement needed to acquire the
  location.
• Remapping allows humans and monkeys to perform a
  spatial memory task accurately.

16
(No Transcript)
17
LIP
V3A
FEF
V3
V2
V1
SC
LGN
Oculomotor System
Retina
18
Stimulus appears outside of RF
Saccade moves RF to stimulus location
19
Stimulus alone control
Saccade alone control
20
Single step task
21
(No Transcript)
22

• Extrastriate Summary
• Remapping occurs at early stages of the visual
  hierarchy.

23

• Extrastriate Summary
• Remapping occurs at early stages of the visual
  hierarchy.
• Corollary discharge has an impact far back into
  the system.

24

• Extrastriate Summary
• Remapping occurs at early stages of the visual
  hierarchy.
• Corollary discharge has an impact far back into
  the system.
• Remapping implies widespread connectivity in
  which many neurons have rapid access to
  information well beyond the classical receptive
  field.

25

• Extrastriate Summary
• Remapping occurs at early stages of the visual
  hierarchy.
• Corollary discharge has an impact far back into
  the system.
• Remapping implies widespread connectivity in
  which many neurons have rapid access to
  information well beyond the classical receptive
  field.
• Vision is an active process of building
  representations.

26

• Remapping in monkey area LIP and extrastriate
  visual cortex
• 2) Remapping in split-brain monkeys
• Behavior
• Physiology
• 3) Remapping in human cortex
• Parietal cortex
• Striate and extrastriate visual cortex

27
Stimulus appears outside of RF
Saccade moves RF to stimulus location
28
What is the brain circuit that produces
remapping?
29
The obvious pathway for visual signals forebrain
commissures
30
Are the forebrain commissures necessary for
updating visual signals across the vertical
meridian? Behavior in double step task
Physiology in single step and double step task
31

32

33
Transfer of visual signals
WITHIN
T2
T1
T2
34
WITHIN
T2
T1
35
WITHIN
T2
T1
T2
T2
T2
36
VISUAL-ACROSS
WITHIN
T2
T1
T2
37
Is performance impaired on visual-across
sequences in split-brain monkeys?
WITHIN
VISUAL-ACROSS
T2
T2
T1
T1
T2
T2
T2
T2
38
(No Transcript)
39
Day 1 Initial impairment for visual-across

Monkey C
40
TRIALS
1-10
41
First day saccade endpoints
Vertical eye position (degrees)
Horizontal eye position (degrees)
42
Last day saccade endpoints
Monkey C
Vertical eye position (degrees)
Monkey E
Monkey E
Horizontal eye position (degrees)
43
Are the forebrain commissures necessary for
updating spatial information across the vertical
meridian?
44
Are the forebrain commissures necessary for
updating spatial information across the vertical
meridian? No. The FC are the primary route but
not the only route.
45
Are the forebrain commissures necessary for
updating spatial information across the vertical
meridian? No. The FC are the primary route but
not the only route. What are LIP neurons doing?
46
Stimulus appears outside of RF
Saccade moves RF to stimulus location
47
(No Transcript)
48
Population activity in area LIP
49
(No Transcript)
50
Split Brain Monkey Summary The forebrain
commissures normally transmit remapped visual
signals across the vertical meridian but they are
not required.
51
Split Brain Monkey Summary The forebrain
commissures normally transmit remapped visual
signals across the vertical meridian but they are
not required. Single neurons in area LIP
continue to encode remapped stimulus traces in
split-brain animals.
52
(No Transcript)
53
(No Transcript)
54

• Remapping in monkey area LIP and extrastriate
  visual cortex
• 2) Remapping in split-brain monkeys
• Behavior
• Physiology
• 3) Remapping in human cortex
• Parietal cortex
• Striate and extrastriate visual cortex

55
(No Transcript)
56
(No Transcript)
57
(No Transcript)
58
(No Transcript)
59
(No Transcript)
60
(No Transcript)
61
(No Transcript)
62
(No Transcript)
63
(No Transcript)
64
Functional Imaging Predictions 1) Robust
activation in cortex ipsilateral to the stimulus.
65

• Functional Imaging Predictions
• 1) Robust activation in cortex ipsilateral to the
  stimulus.
• 2) Ipsilateral activation should be smaller than
  the contralateral visual response.

66

• Functional Imaging Predictions
• 1) Robust activation in cortex ipsilateral to the
  stimulus.
• 2) Ipsilateral activation should be smaller than
  the contralateral visual response.
• 3) It should not be attributable to the stimulus
  alone or to the saccade alone.

67

• Functional Imaging Predictions
• 1) Robust activation in cortex ipsilateral to the
  stimulus.
• 2) Ipsilateral activation should be smaller than
  the contralateral visual response.
• 3) It should not be attributable to the stimulus
  alone or to the saccade alone.
• 4) Ipsilateral activation should occur around the
  time of the saccade.

68
(No Transcript)
69
(No Transcript)
70
(No Transcript)
71
Contralateral Visual Response
72
Ipsilateral Remapped Response
73
Ipsilateral Remapped Response
74
(No Transcript)
75
(No Transcript)
76
Visual and Remapped Responses
77
(No Transcript)
78
(No Transcript)
79
(No Transcript)
80
(No Transcript)
81
(No Transcript)
82
(No Transcript)
83
(No Transcript)
84
(No Transcript)
85

• Human Parietal Imaging Summary
• Remapping in humans produces activity in parietal
  cortex ipsilateral to the visual stimulus.

86

• Human Parietal Imaging Summary
• Remapping in humans produces activity in parietal
  cortex ipsilateral to the visual stimulus.
• Remapped activity is lower amplitude than visual
  activity.

87

• Human Parietal Imaging Summary
• Remapping in humans produces activity in parietal
  cortex ipsilateral to the visual stimulus.
• Remapped activity is lower amplitude than visual
  activity.
• It cannot be attributed to the stimulus or the
  saccade alone.

88

• Human Parietal Imaging Summary
• Remapping in humans produces activity in parietal
  cortex ipsilateral to the visual stimulus.
• Remapped activity is lower amplitude than visual
  activity.
• It cannot be attributed to the stimulus or the
  saccade alone.
• It occurs in conjunction with the eye movement.

89

• Remapping in monkey area LIP and extrastriate
  visual cortex
• 2) Remapping in split-brain monkeys
• Behavior
• Physiology
• 3) Remapping in human cortex
• Parietal cortex
• Striate and extrastriate visual cortex

90
(No Transcript)
91
(No Transcript)
92
(No Transcript)
93
(No Transcript)
94
(No Transcript)
95
(No Transcript)
96
(No Transcript)
97
(No Transcript)
98
Contralateral Visual Response
99
Ipsilateral Remapped Response
100
Remapping in Multiple Visual Areas
101

• Remapping in monkey area LIP and extrastriate
  visual cortex
• 2) Remapping in split-brain monkeys
• Behavior
• Physiology
• 3) Remapping in human cortex
• Parietal cortex
• Striate and extrastriate visual cortex

102

• Human Imaging Summary
• Remapping in humans produces activity in the
  hemisphere ipsilateral to the stimulus.

103

• Human Imaging Summary
• Remapping in humans produces activity in the
  hemisphere ipsilateral to the stimulus.
• Remapped activity is present in human parietal,
  extrastriate and striate cortex.

104

• Human Imaging Summary
• Remapping in humans produces activity in the
  hemisphere ipsilateral to the stimulus.
• Remapped activity is present in human parietal,
  extrastriate and striate cortex.
• Remapped visual signals are more prevalent at
  higher levels of the visual system hierarchy.

105

• Human Imaging Summary
• Remapping in humans produces activity in the
  hemisphere ipsilateral to the stimulus.
• Remapped activity is present in human parietal,
  extrastriate and striate cortex.
• Remapped visual signals are more prevalent at
  higher levels of the visual system hierarchy.
• Remapping occurs in parietal and visual cortex.

106
Conclusions Remapping of visual signals is
widespread in monkey cortex.
107

• Conclusions
• Remapping of visual signals is widespread in
  monkey cortex.
• Split-brain monkeys are able to remap visual
  signals across the vertical meridian.

108

• Conclusions
• Remapping of visual signals is widespread in
  monkey cortex.
• Split-brain monkeys are able to remap visual
  signals across the vertical meridian.
• Remapped visual signals are present in area LIP
  in split-brain monkeys.

109

• Conclusions
• Remapping of visual signals is widespread in
  monkey cortex.
• Split-brain monkeys are able to remap visual
  signals across the vertical meridian.
• Remapped visual signals are present in area LIP
  in split-brain monkeys.
• Remapped visual signals are robust in human
  parietal and visual cortex.

110

• Conclusions
• Remapping of visual signals is widespread in
  monkey cortex.
• Split-brain monkeys are able to remap visual
  signals across the vertical meridian.
• Remapped visual signals are present in area LIP
  in split-brain monkeys.
• Remapped visual signals are robust in human
  parietal and visual cortex.
• Vision is an active process of building
  representations from sensory, cognitive and motor
  signals.

111
(No Transcript)
112
(No Transcript)
113
Learning? Or a monkey trick?
114
no monkey tricks..
115
Both monkeys really update the visual
representation
116
(No Transcript)
117
(No Transcript)
118
Magnitude of Remapped Response
119
(No Transcript)
120
(No Transcript)
121
(No Transcript)
122
(No Transcript)
123
(No Transcript)
124
(No Transcript)
125
Intact Subjects
Split Brain Subject
126
(No Transcript)
127
(No Transcript)
128
Strength of Parietal Responses in Split Brain
and Intact Subjects