Why do things happen the way they do in your imagination?

1
Why do things happen the way they do in your
imagination?

• Is it because of the format of your image or your
  cognitive architecture? Or because of what you
  know?
• Did it reveal a capacity of mind?
• Or was it because you made it do what it did?
• Can you make your image have any properties you
  choose? Or behave in any way you want? Why not?
• How about imagining an object from all directions
  at once, or from no particular direction?
• How about imagining a 4-dimensional object?
• Can you imagine a printed letter which is neither
  upper nor lower case? A triangle that is not a
  particular type?

2
More on Representation of space and arguments
from neuroscience

• The intuition that images are spatial (or, as
  some put it, that they have space) is an
  interesting one and deeper than most other
  questions being studied in laboratories. I have
  discussed this in excruciating detail in the
  last chapter of my Things and Places.
• According to Kosslyn, the argument has now
  entered the third and final stage, where
  neuroscience evidence has put an end to the
  debate. I will look at some of this alleged
  debate-ending evidence and will show that it is
  characterized by desperation and lack of critical
  analysis.

3
A brief look at whether we may be storing
information in the form of an icon

• For present purposes I want to look at the issue
  of explanation and I will do this by considering
  the sense in which spatial information may
  (must?) itself be spatial
• This matters to the question of what forms of
  representation we have in our minds and how these
  refer to things in the world.

4
Our studies of mental scanning
(Pylyshyn Bannon. See Pylyshyn, 1981)
There is even reason to doubt that one can
imagine scanning continuously (Pylyshyn Cohen,
1998)
5
Does visual imagery use the visual system?
Major Question 2

• It depends on what you mean by use and visual
  system
• To use vision can mean actually to perceive the
  image
• Only early vision is relevant to this use of
  perceive, since general vision can involve all
  of cognition
• This question is of interest to picture theorists
  because if vision is involved it may suggest that
  images are uninterpreted picture-like (or
  depictive) spatial displays. The visual module
  cannot be applied to already-interpreted data
  structures
• Lets assume that the visual system is active
  during mental imagery as some have suggested
• What follows from that?
• Why should we believe that vision may be
  involved?
• In the last 15 years the main support for the
  assumption that vision is involved has come from
  neuroscience.

6
Reasons for thinking that images are interpreted
by the visual system

• Similar phenomenology of imagining seeing
• This reason overshadows all others
• Superposition interference studies
• Visual illusions with projected images
• The ubiquitous role of attention
• Re-perceiving and novel construals
• A large but very problematic literature

7
More demonstrations of the relation between
vision and imagery

• Images constructed from descriptions
• The D-J example(s)
• The two-parallelogram example
• Amodal completion
• Reconstruals Slezak

8
Can images be visually reinterpreted?

• There have been many claims that people can
  visually reinterpret images
• These have all been cases where one could easily
  figure out what the combined image would look
  like without actually seeing it (e.g., the J D
  superposition).
• Pedersons careful examination of visual
  reconstruals showed (contrary to her own
  conclusion) that images are never ambiguous (no
  Necker cube or figure-ground reversals) and when
  new construals were achieved from images they
  were quite different from the ones achieved in
  vision (more variable, more guessing from cues,
  etc).
• The best evidence comes from a philosopher
  (Slezak, 1992, 1995)

9
Slezak figures
Pick one (or two) of these animals and memorize
what they look like. Now rotate it in your mind
by 90 degrees clockwise and see what it looks
like.
10
Slezak figures rotated 90o
11
Do this imagery exerciseImagine a parallelogram
like this one
Connect each corner of the top parallelogram with
the corresponding corner of the bottom
parallelogram
Now imagine an identical parallelogram directly
below this one
What do you see when you imagine the connections?
Did the imagined shape look (and change) like the
one you see now?
12
Amodal completion by images?
13
Is this what you saw?
14
Off-retinal info different from foveal info
15
Off-retinal info different from foveal info
16
Labels propagate over picture
17
Anorthoscope Scan view How many contours are
there?
18
Anorthoscope Scan view What is the shape?
19
Vision is involved when images are superimposed
onto vision

• Many experiments show that when you project an
  image onto a display the image acts very much
  like a superimposed display
• Shepard Podgorny, Hayes,
• Interference effects (Brooks)
• Mental scanning
• Interaction with the motor system (Finke)

20
Shepard Podgorny experiment
Both when the displays are seen and when the F is
imagined, RT to detect whether the dot was on the
F is fastest when the dot is at the vertex of the
F, then when on an arm of the F, then when far
away from the F and slowest when one square off
the F.
21
Brooks spatial interference study
Respond by pointing to symbols in a table or by
saying the words left or right
22
Visual-Motor adaptation and motor adaptation to
images

• The basic prism adaptation setup arm movement
  towards a target while wearing prism glasses
• Now repeat with arm unseen but subject told where
  it is (actually where it would have been in the
  prism case)
• Get adaptation Finke, R. A. (1979). The
  Functional Equivalence of Mental Images and
  Errors of Movement. Cognitive Psychology, 11,
  235-264.
• But in the original experiment you dont need to
  see a hand, any indicator of where the hand is
  will do as long as the subject believes it is
  where indicated

23
Standard view of saccadic integration by
superposition
Is it plausible? Is it true?
24
Superposition does not work (Oregan 1983)
25
This is what our conscious experience suggests
goes on in vision
Kliban
26
This is what the demands of explanation suggests
must be going on in vision
27
Are there pictures in the brain?

• There is no evidence for cortical displays of the
  right kind to explain visual or imaginal phenomena

28
Is there very short iconic storage?

• Although the idea of pictorial long-term memory
  is not supported, there is some provisional
  evidence that sensory information outlasts the
  duration of the stimulus. Many people have
  studied these sensory buffers including George
  Sperling and Michael Posner.

29
Sperlings partial report method for showing an
iconic memory
30
Posners demonstration of short lasting shape
information
Fast
Slower
Fast
Slower
31
So what does the visual system produce?

• The best hypothesis so far (i.e., the only one
  that has not been shown to be clearly on the
  wrong track) is that the brain is a species of
  computer in which representations of the world
  are encoded in the form of symbol structures, and
  actions are determined by calculations (i.e.,
  inferences) based on these symbolic encodings.
• Therefore vision (and other sensory systems)
  compute symbolic expression for the rest of the
  (Turing) computing machinery to use.

32
So why does it not feel like we are doing
computations?

• The content of our conscious experience is a very
  poor guide to what is actually going on that
  causes both our experiences and our behavior.
  Science is concerned with causes, not just
  correlations.
• We have learned (since Galileo) that we cant
  assume that the way things seem has much to do
  with how it works (consider the example of
  language understanding)
• As in most sciences, the essential causes are far
  from obvious (e.g., How can the moon exert a pull
  on the earth without contacting it? What is this
  table made of? etc.).
• In the case of cognition, what is going on is a
  delicate mixture of the obvious (what Granny or
  Shakespeare knew about people and why they do
  what they do) and the incredible.

33
We cant even be sure that we have the right
methods or instruments
34
Mental Scanning

• Some hundreds of experiments have now been done
  demonstrating that it takes longer to scan
  attention between places that are further apart
  in the imagined scene. In fact the relation is
  linear between time and distance.
• These have been reviewed and described in
• Denis, M., Kosslyn, S. M. (1999). Scanning
  visual mental images A window on the mind.
  Cahiers de Psychologie Cognitive / Current
  Psychology of Cognition, 18(4), 409-465.
• Rarely cited are experiments by Pylyshyn Bannon
  which I will summarize for you.

35
Studies of mental scanningDoes it show that
images have metrical space?
(Pylyshyn Bannon. See Pylyshyn, 1981)

• Conclusion The image scanning effect is
  Cognitively Penetrable
• i.e., it depends on goals and beliefs, or on
  Tacit Knowledge.

36
What is assumed in imagist explanations of mental
scanning?

• In actual vision, it takes longer to scan a
  longer distance because real distance, real
  motion, and real time is involved, therefore this
  equation holds due to natural law
• Time distance speed
• But what ensures that a corresponding relation
  holds in an image? The obvious answer is
  Because the image is laid out in space!
• But what if that option is closed for empirical
  reasons?
• Imagists appeal to a Functional Space which
  they liken to a matrix data structure in which
  some cells are adjacent to other cells, some are
  closer and others further away, and to move from
  one to another it is natural that you pass
  through intermediate cells
• Question What makes these sorts of properties
  natural in a matrix data structure?

37
Thou shalt not cheat

• There is no natural law that requires the
  representations of time, distance and speed to be
  related according to the motion equation. You
  could equally easily imagine an object moving
  instantly or according to any motion relation you
  like, since it is your image!
• There are two possible answers why the relation
• Time Representation of distance
  Representation of speed
• typically holds in an image-scanning task
• Because subjects have tacit knowledge that this
  is what would happen if they viewed a real
  display, or
• Because the matrix is taken to be a simulation of
  a real-world display, as it often is in computer
  science

38
Mental Rotation has been one of the most cited
demonstrations of all

• Look at the following 3D figures and judge which
  pairs are the same except for orientation. The
  other pair are enantiomorphs 3D mirror images
  so they cant be put into correspondence by 3D
  rotation only.

39
Mental rotation
Time to judge whether (a)-(b) or (b)-(c) are the
same except for orientation increases linearly
with the angle between them (Shepard Metzler,
1971)
40
What do you do to judge whether these two figures
are the same shape?
Is this how the process looked to you?
When you make it rotate in your mind, does it
seem to retain its rigid 3D shape without
re-computing it?
41
Thou shalt not cheat

• What happens in ALL imagist accounts of phenomena
  from mental scanning to mental rotation is that
  they assume the properties of real space in order
  to provide a principled explanation, then retreat
  to something not-quite-real space when it is
  pointed out that they are assuming that images
  are laid out in real space-in-the-head.
• This happens with mental rotation as well, even
  though the tacit knowledge account is not
  plausible there (it is an involuntary and
  universal way of solving the rotated-figure task
  so long as the task involves tokens of
  enantiomorphs).

42
The missing bit of logic

• According to Prinz (2002) p 118,If visual-image
  rotation uses a spatial medium of the kind
  Kosslyn envisions, then images must traverse
  intermediate positions when they rotate from one
  position to another. The propositional system
  can be designed represent intermediate positions
  during rotation, but that is not obligatory.
• Given that this happens in 3D so that it cant be
  a literal brain space, the question arises What
  makes this obligatory in functional Space?

43
Sources of obligatory constraint

• It could be that there is a medium or a set of
  analog properties that together happen to
  simulate a virtual space, and the physical
  properties of this medium enforce the continuity
  of motion through it. This is very unlikely.
• It could be that the constraint comes from the
  fact that it holds in the world. Then its
  transfer from the real to the mental world occurs
  either
• Voluntarily, because we know how it would happen
  and we can use that fact to solve the problem
• Naturally, because the constraint got built in
  over time through evolutionary pressures
• Naturally, but not because the constraint is
  built in, but because of other properties of the
  architecture that make it more efficient to
  compute the rotated shape incrementally until
  there is a match

44
Do images have low-level psychophysical
properties?

• Bring the bars closer and closer together. In
  which can you see the bars when the spacing is
  closest?
• In experiments, it was shown that the oblique
  effect occurs in mental images just as it does in
  vision
• Kosslyn, Thompson Ganis (2006) argued that this
  is because there are more vertical and
  horizontally tuned cells than obliquely tuned
  cells in visual cortex. Does this explain the
  finding?

45
Do images have low-level visual properties?

• Imagine a grating in which the bars are
• Horizontal
• Vertical
• Oblique (45)

46
A final point

• In Kosslyn, Thompson Ganis (2007) the authors
  cite Ned Block to the effect that one does not
  need an actual 2D surface, so long as the
  connections upstream from the cortical surface
  can decode certain pairs of neurons in terms of
  their imagined distance. Imagine long stretchy
  axons going from a 2D surface to subsequent
  processes. Now imagine that the neurons are
  randomly moved around so they are no long on a 2D
  layout. As long as the connections remain fixed
  it will still behave as though there was a 2D
  surface.
• Call this the encrypted 2D layout version of
  literal space.

47
The encrypted-spatial layout alternative

• By itself the encrypted-layout alternative will
  not do because without referring to the original
  2D locations, the relation between pairs of
  neurons and scan time is not principled. In the
  end the only principle we have is
  Timedistance/speed so unless the upstream system
  decrypts the neuron locations into their original
  2D surface locations the explanation for the
  increase in time with increased imagined distance
  remains a mere stipulation. It stipulates, but
  does not explain why, when two points are further
  away in the imagined layout it takes longer to
  scan between them or why scanning between them
  requires that one visit intermediate locations
  along the way.
• But this is what we needed to explain! One can
  apply such a mere stipulation to any form of
  representation. What was a principled
  explanation with the literal 2D display has now
  been given up for a mere statement of how it
  shall be.

48
The Imagery Debate Redux

• According to Kosslyn there have been 3 stages in
  the debate over the nature of mental images
• The role of images in learning and memory
  (Paivios Dual Code theory). Influential at the
  time but now abandoned except for a few
  recidivists like Barsalou.
• Spatial properties of images and their role in
  dynamic processes, as assessed by reaction time
  measures (Kosslyns research on metric
  properties of images)
• Discovery of brain mechanisms underlying visual
  imagining, which led to the resolution of the
  imagery debate.

49
Mental imagery and neuroscience

• Neuroanatomical evidence for a retinotopic
  display in the earliest visual area of the brain
  (V1)
• Neural imaging data showing V1 is more active
  during mental imagery than during other forms of
  thought
• The form of activity differs for small vs large
  images in the way that it differs when viewing
  small and large displays
• Transcranial magnetic stimulation of visual areas
  interferes more with imagery than other forms of
  thought
• Clinical cases show that visual and image
  impairment tend to be similar (Bisiach, Farah)
• More recently psychophysical measures of images
  shows parallels with comparable measures of
  vision, and these can be related to the receptive
  cells in V1

50
Neuroscience has shown that the retinal pattern
of activation is displayed on the surface of the
cortex
There is a topographical projection of retinal
activity on the visual cortex of the cat and
monkey.
Tootell, R. B., Silverman, M. S., Switkes, E.,
de Valois, R. L. (1982). Deoxyglucose analysis of
retinotopic organization in primate striate
cortex. Science, 218, 902-904.
51
Problems with drawing conclusions about the
nature of mental images from neuroscience data

• The capacity for imagery and for vision are known
  to be independent. Also all imagery results are
  observed in the blind.
• Cortical topography is 2-D, but mental images are
  3-D all phenomena (e.g. rotation) occur in
  depth as well as in the plane.
• Patterns in the visual cortex are in retinal
  coordinates whereas images are in
  world-coordinates
• Your image stays fixed in the room when you move
  your eyes or turn your head or even walk around
  the room
• Accessing information from an image is very
  different from accessing it from the perceived
  world. Order of access from images is highly
  constrained.
• Conceptual rather than graphical properties are
  relevant to image complexity (e.g., mental
  rotation).

52
Problems with drawing conclusions about mental
images from the neuroscience evidence

1. Retinal and cortical images are subject to
   Emmerts Law, whereas mental images are not
2. The signature properties of vision (e.g.
   spontaneous 3D interpretation, automatic
   reversals, apparent motion, motion aftereffects,
   and many other phenomena) are absent in images
3. A cortical display account of most imagery
   findings is incompatible with the cognitive
   penetrability of mental imagery phenomena, such
   as scanning and image size effects
4. The fact that the Minds Eye is so much like a
   real eye (e.g., oblique effect, resolution
   fall-off) should serve to warn us that we may be
   studying what observers know about how the world
   looks to them, rather than what form their images
   take.

53
Problems with drawing conclusions about mental
images from the neuroscience evidence

• Many clinical cases can be explained by appeal to
  tacit knowledge and attention
• The tunnel effect found in vision and imagery
  (Farah) is likely due to the patient knowing what
  things now looked like to her post-surgery
• Hemispatial neglect seems to be a deficit in
  attention, which also explains the
  representational neglect in imagery reported by
  Bisiach
• A recent study shows that imaginal neglect does
  not appear if patients have their eyes closed.
  This fits well in the account I will offer in
  which the spatial character of a mental images
  derives from concurrently perceived space.
• What if colored three-dimensional images were
  found in visual cortex? What would that tell you
  about the role of mental images in reasoning?
  Would this require a homunculus?

54
Should we welcome back the homunculus?

• In the limit if the visual cortex contained the
  contents of ones conscious experience in imagery
  we would need an interpreter to see this
  display in visual cortex
• But we will never have to face this prospect
  because many experiments (including ones by
  Kosslyn) show that the contents of mental images
  are conceptual (or, as Kosslyn puts it, contain
  predigested information).
• And finally, it is clear to anyone who thinks
  about it for a few seconds that you can make your
  image do whatever you want and to have whatever
  properties you wish.
• There are no known constraints on mental images
  that cannot be attributed to lack of knowledge of
  the imagined situation (e.g., imagining a 4D
  cube).
• All currently claimed properties of mental images
  are cognitively penetrable.

55
Explaining mental scanning, mental rotation and
image size effects in terms of functional space

• When people are faced with the natural conclusion
  that the iconic position entails space (as in
  scanning and size effects) they appeal to
  functional space
• A Matrix in a computer are often cited as an
  example
• Consider a functional space account of scanning
  or of mental rotation
• Why does it take longer to scan a greater
  distance in a functional space?
• Why does it take longer to rotate a mental image
  a greater angle?

56
But there are examples of solving geometry
problems easily with imagery

• There are many problems that you can solve much
  more easily when you imagine a layout than when
  you do not.
• In fact many instances of solving problems by
  imagining a layout that seem very similar to how
  would solve them if one had pencil-and-paper.
• The question of how pictures, graphs, diagrams,
  etc help in reasoning is very closely related to
  the question of how imagined layouts function in
  reasoning. That is not in question. What is in
  question is what happens in either the visual or
  imagined cases and how images can benefit from
  this processes even though there is no real
  diagram.

57
How do real visual displays help thinking?

• Why do diagrams, graphs, charts, maps, icons and
  other visual objects help us to reason and to
  solve problems?
• The question why visual aids help is nontrivial
  and Seeing Visualizing, chapter 8 contains some
  speculative discussion, e.g., they allow the
  visual system to
• make certain kinds of visual inferences
• make use of visual demonstratives to offload some
  of the memory load
• Capitalize on the fact that the displays embody
  the axioms of measure theory and of geometry
  (which are then inherited by thought)
• The big question is whether any of these
  advantages carry over to imaginal thinking! Do
  mental images have some (or any) of the critical
  properties that make diagrams helpful in
  reasoning?

58
Visual inferences?

• If we recall a visual display it is because we
  have encoded enough information about its
  visual-geometrical properties that we can meet
  some criteria, e.g., we can draw it. But there
  are innumerably many ways to encode this
  information that are sufficient for the task
  (e.g. by encoding pairwise spatial relations,
  global spatial relations, and so on). For many
  properties the task of translating from one form
  to another is much more difficult than the task
  of visually encoding it the translation
  constitutes visual inference.
• The visual system generalizes from particular
  instances as part of its object-recognition skill
  (all recognition is recognition-as and therefore
  assumes generalization from tokens to types). It
  is also very good at noticing certain properties
  (e.g., relative sizes, deviations from square or
  circle, collinearity, inside, and so on). These
  capabilities can be exploited in graphical
  layouts.

59
Memorize this map so you can draw it accurately
60
From your memory

• Which groups of 3 or more locations are
  collinear?
• Which locations are midway between two others?
• Which locations are closest to the center of the
  island?
• Which pairs of locations are at the same
  latitude?
• Which is the top-most (bottom-most) location?
• If you could draw the map from memory using
  whatever properties you noticed and encoded, you
  could easily answer the questions by looking at
  your drawing even if you had not encoded the
  relations in the queries.

61
Draw a rectangle. Draw a line from the bottom
corners to a point on the opposite vertical side.
Do these two lines intersect? Is the point of
intersection of the two lines below or above the
midpoint? Does it depend on the particular
rectangle you drew?
B
A
C
D
62
Which properties of a real diagram also hold for
a mental diagram?

• A mental diagram does not have any of the
  properties that a real diagram gets from being on
  a rigid 2D surface.
• When you imagine 3 points on a line, labeled A,
  B, and C, must B be between A and C? What makes
  that so? Is the distance AC greater than the
  distance AB or BC?
• When you imagine drawing point C after having
  drawn points A and B, must the relation between A
  and B remain unchanged (e.g., the distance
  between them, their qualitative relation such as
  above or below). Why?
• These questions raise what is known as the frame
  problem in Artificial Intelligence. If you plan
  a sequence of actions, how do you know which
  properties of the world a particular action will
  change and which it will not, given that there
  are an unlimited number of properties and
  connections in the world?

63
One view Images as depictive representations

• A depictive representation is a type of picture,
  which specifies the locations and values of
  configurations of points in a space. For
  example, a drawing of a ball on a box would be a
  depictive representation.
• The space in which the points appear need not be
  physical, but can be like an array in a
  computer, which specifies spatial relations
  purely functionally. That is, the physical
  locations in the computer of each point in an
  array are not themselves arranged in an array it
  is only by virtue of how this information is
  read and processed that it comes to function as
  if it were arranged into an array (with some
  points being close, some far, some falling along
  a diagonal, and so on).
• Depictive representations convey meaning via
  their resemblance to an object, with parts of the
  representation corresponding to parts of the
  object When a depictive representation is used,
  not only is the shape of the represented parts
  immediately available to appropriate processes,
  but so is the shape of the empty space
  Moreover, one cannot represent a shape in a
  depictive representation without also specifying
  a size and orientation.
  (Kossyln, 1994, p 5)

64
If all else fails there is always parsimony and
generality(they worked well in physics and
linguistics!)