Neural Visuomotor Controller for a Simulated Salamander Robot - PowerPoint PPT Presentation


PPT – Neural Visuomotor Controller for a Simulated Salamander Robot PowerPoint presentation | free to view - id: 74481-ZDc1Z


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation

Neural Visuomotor Controller for a Simulated Salamander Robot


Neural Visuomotor Controller for a Simulated Salamander Robot. Biljana Petreska ... The salamander has difficulties with following the prey stimulus as the amount ... – PowerPoint PPT presentation

Number of Views:47
Avg rating:3.0/5.0
Slides: 41
Provided by: petr5


Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: Neural Visuomotor Controller for a Simulated Salamander Robot

Neural Visuomotor Controller for a Simulated
Salamander Robot
  • Biljana Petreska
  • Diploma Thesis March 2004
  • Responsible
  • Prof. Auke Jan Ijspeert

Goals of the Project
  • Investigate through simulations tightly coupled
    with neurobiological data, the neural mechanisms
    underlying visually guided behaviour in
  • Implement a closed-loop with the environment onto
    the existing neuromechanical simulation developed
    by Ijspeert, by adding biologically inspired
    models for parts of the salamander brain
  • Develop a controller that accounts for
    observations in feeding behaviour, including prey
    localization and prey recognition
  • Study a model proposed by Ijspeert of structured
    mapping between the optic tectum (primary visual
    processing center) and the brain stem (motor
    centers) as a solution to the visuomotor

  • Relevant for perceptual robotics
  • decoding the brain processes, assigning meaning
    to complex patterns of sensor stimuli may lead to
    the solution of many robotics tasks
  • Test bed for probing neurobiological
  • ideal for the validation or refutation of new

  • Short introduction on relevant topics and
    previous works
  • Implemented Models
  • Respective Results
  • Conclusion and Future Work

Everything youve always wanted to know on
  • Amphibians
  • Great variety of species (3924 indexed so far),
    sizes (from 16mm to 1.5m), aspects and lifestyles
    (terrestrial and/or aquatic).
  • A relatively simple neural circuitry that
    presents all main vertebrate features
  • Tractable from an experimental point of view an
    important amount of behavioral, biological and
    neurological data exists

Visually Guided Behavior
  • Vision is by far the most important feeding
    guiding sense. Under good visual conditions the
    other signals such as olfactory are overridden
  • Feeding strategies (some species can switch from
    one to another)
  • hunter strategy active search for prey.
    Prerequisites are a short massive tongue and poor
    visual capacities.
  • ambush strategy wait until prey comes close.
    Prerequisites are a highly specialized projectile
    tongue (up to 80 body length), evolved visual
    system and frontally oriented eyes

Visually Guided Behaviour
  • Sequence of feeding behavior
  • orienting
  • approach
  • olfaction tests
  • gaze stabilization
  • snapping
  • Prey preferences (in order of importance)
  • stimulus size
  • stimulus velocity
  • stimulus-background contrast
  • stimulus shape
  • movement pattern
  • experience-dependant

Morphology of the Salamander Brain
  • Functional differentiation of the brain
    structurally different regions accomplish
    different tasks
  • Global top to bottom visual information
  • Principal components photoreceptors, retina,
    optic tectum, nucleus isthmi, pretectum,thalamus,
    medulla oblongata and brain stem

Retinal Ganglion Cells
  • First layer of visual processing, transfers
    visual signals to the brain via the optic nerve
  • 3 Types of retinal ganglion cells that project to
    particular layers in the optic tectum

Optic Tectum
  • Main visual processing center. Integrates also
    multimodal perception, such as ascending
    somatosensory, auditory, olfaction and vestibular
  • Stratification in 9 layers, first three are
    retinal afferents
  • Six morphological neuron types identified (one
  • Topographic representation of the visual field
  • Viewed as a set of partial overlapping maps due
    to the different types of tectal projection
  • Number of tectal cells in Hydromantes Italicus
    92 000 and 3300 out of 5000 projection neurons
    are descending
  • Projection patterns from and to the retina,
    pretectum, thalamus, nucleus isthmi and medulla
    (reaching the spinal cord)

Distribution and Receptive Field Sizes of Tectal
Neurons in H.Italicus
  • Has been ascribed a role in optokinetic
    nystagmus, figure-background discrimination,
    pupillary reflex, fixation, phototaxis, and
    prey-enemy distinction.
  • Properties of pretectal neurons
  • Homogenous arborisation
  • (no classification was possible)
  • Divergent projections
  • (including to the tectum and spinal cord)
  • Large receptive fields
  • Receive direct and indirect (from tectum) retinal
  • Direction-sensitive neurons (predominantly in
    temporonasal direction)
  • Respond to stimuli in the contralateral visual

Lesion Experiments
  • Give insight of the function of the destroyed
    brain region
  • Lesion of the optic tectum both visual
    prey-catching and predator avoidance fail to
    occur. Local lesions produce scotoma, total
    blindness for a part of the visual field
    corresponding to the size of the lesion.
  • Lesion of the pretectum locally facilitates
    feeding and abolishes prey-predator
    discrimination, attack everything that moves
    including their own extremities and threatening
  • Lesion of the thalamus unable to avoid collision
    to a vertically stripped barrier, affects the
    binocular field
  • Lesion of the medulla oblongata affects
    distance, elevation or horizontal eccentricity
    estimates, overshoots prey or snaps only in
    frontal positions gt different components of the
    stimulus position are handled through different
  • Difficulty in some cases the animal recovers
    shortly after the lesion and the relative
    precision of lesions may induce errors

Previous Works
  • Based upon the principle of coarse coding (Eurich
    et al, 1997)
  • Motivation the high sensory resolution observed
    in nature seems incompatible with the large size
    of receptive fields of tectal neurons
  • Definition population-coding using mapping
    combinatorics of intersecting receptive fields
  • A non-firing neuron conveys as much information
    as a firing neuron. All neurons participate at
    the information coding
  • Weakness likely to suffer from metamery
    (convergence of information channels)
  • Simulander I
  • Feedforward network with only 100 neurons,
    trained by an evolution strategy for the specific
    task of head orienting (implies prey
  • Distribution and sizes of receptive fields of
    tectal neurons are respected and firing rates
    have been adapted
  • Unstructured mapping follows the prey with high
  • Simulander II
  • Similar to Simulander I, but trained for the
    specific task of frontal tongue projection
    (implies depth perception)

Addressed Questions
  • How can the stimulus location and depth estimates
    be extracted from the tectum maps?
  • What sensorimotor transformations occur at the
    level of the optic tectum, the brainstem and the
    pathways between them? Can a structured mapping
    provide an accurate visual tracking?
  • Which type of a tectum-brainstem mapping explains
    the typical curved approach in monocularized
  • How is the visual perception influenced by head
    motion during the approach toward a stimulus? Are
    additional mechanisms necessary for dealing with
    the remaining shifts in the visual background?
  • Which mechanism implements the release of the
    snapping behavior? And how is the tongue

Neural Networks
  • Restrictions (performance motivated)
  • Uniformly distributed neuron units
  • Square receptive fields
  • Specification
  • Center receptive field (in degrees of visual
    field) determines the size of the neural
  • Surround receptive field (in degrees of visual
    field) determines the overlap and redundancy
  • Weights matrix, activation function and
  • Features
  • Reduction (biologically motivated)
  • Visualization (extremely practical)

Eyes of the Simulated Salamander
  • Virtual cameras extract views using provided
    OpenGL functions
  • Correct the view using a spherical projection
  • Photoreceptors are equivalent to pixel grey
  • Scalable visual field

Retinal Ganglion Cells of Type 1
  • Properties
  • small size excitatory (2-3) and strong
    inhibitory (12-16) receptive fields
  • no response to change in light
  • involved in local contrast calculation gt edge
  • project to the contralateral spinal cord gt
    obstacle avoidance?
  • give rise to a fine grained representation of the
    visual field in the retina
  • Modelled with the laplacian of a gaussian filter
  • Classic edge detector in computer vision and
    confirmed by the study of a larval tiger
    salamander retina receptive field

Retinal Ganglion Cells of Type 2
  • Type 2 retinal ganglion cells respond only to
    moving objects gt motion detectors
  • Detection of change compare the corresponding
    pixels at different times, using a linear
    difference function
  • where t is a threshold, j and k are moments in
    time, x and y are the pixel positions in the
  • Biological inspirations
  • Reflects signals with delayed pathways that give
    rise to a simultaneous representation of the same
    object at different times in the brain
  • Flat weights the activity sharply increases when
    an object enters the receptive field variation
  • Tectal neurons are contrast-sensitive linear

Retinal Ganglion Cells of Type 3
  • Properties
  • large receptive fields (10-20)
  • tonic response to change in light intensity
  • respond to overall luminosity (dimming detectors)
  • respond also at low contrast and velocity
  • Model
  • flat weights
  • simple summing network
  • Predator detectors among other

Optic Tectum Model
  • Principal biological inspirations
  • Retinotopic map in the optic tectum electrical
    stimulations result in turning movements that
    roughly correspond to this map
  • Only two synapses between the retina and the
    brain stem the tectum directly projects onto the
    brain stem
  • Input retinal ganglion cells of type 2. Motion
    is a necessary prerequisite for a stimulus to be
    interpreted as prey
  • Structured mapping different strengths along the
    rostro-caudal axis, reflects the stimulus
  • Motoneuron activation function (integrating
    weighted tectal activity)
  • where x is the change in light intensity of the
    pixel at positions i and j
  • Linear weights function
  • where a and ß are parameters

Optic Tectum Model II
  • Version with ipsilateral input
  • (contribution from both eyes)

Optic Tectum Model III
  • Normalizing tectal activity. The modified model
    is robust to changes in the stimulus parameters
    and visual scene
  • Biological reference
  • TO4 neurons, arborize in RGC2
  • TO2 neurons, large receptive fields
  • both project to the nucleus isthmi

Pretectum Model
  • Large stimuli based on RGC3 gt dimming detectors
    with three times larger receptive fields
  • Motion compare direct and indirect (via tectum)
    RGC3 responses
  • Direction-sensitive neurons
  • Why temporonasal sensitivity?
  • Based upon separating the ON and OFF channels
  • Hypothesis only sensitive to dark objects
    (biologically consistent)

Snapping Model
  • Relevant for depth estimation
  • Tongue mechanism (biologically consistent) 4
    muscles, protraction and retraction times
    modulated by the stimulus position
  • We proposed a mechanism for frontal snapping
    based upon divergent projections of the tectal

  • Find optimal a and ß parameters of the linear
    weights function through an exhaustive search of
    the parameter space
  • Cost function difference between the stimulus
    direction and the salamander orienting movement
  • Experimental conditions
  • Ewert experiment the stimulus is moved on a
    semi-circular trajectory with a constant speed in
    front of the animal
  • (task of head orienting)
  • Body muscles were inhibited (only neck muscles)
  • The stimulus parameters (size, speed, distance,
    ...) and network parameters (number of neurons)
    were fixed according to values found in
    literature. Both single stimulus and complex
    background were used

Optimal Values
  • Many combinations of values give similar results
  • Good results are also achieved without
    ipsilateral input

a and ß Parameters
  • Regular parameter space. With different fixed
    values the aspect is conserved and the minimal
    error area (in black) is shifted
  • ß parameters are not essential, the minimal error
    area is centered in point (0,0)

Performance Results
  • An accuracy of less than 3 (real value) is
    achieved for small stimulus velocity values with
    20000 RGC2 and 2000 (less than 3300) tectal
  • Robustness stable reaction to change in
    stimulus parameters and visual scene

With Complex Background
  • The salamander has difficulties with following
    the prey stimulus as the amount of noise is
    considerable. It discriminates between objects
    with same apparent angular size, however orients
    at "average flies"
  • The model should be coupled with a selective
    visual attention mechanism (enhanced retinal
    signals in the area containing the prey stimulus)
    and/or optokinetic or vestibucollic image
    stabilization reflexes (antagonistic head
    movements that compensate for body undulations)
  • Integrating approach is trivial with a unique
    prey stimulus

  • The salamander discriminates between a small prey
    object and a large predator object
  • When the pretectum is abolished, escape behavior
    fails to occur
  • Delayed response the salamander escapes for a
    longer time than the predator is visible
  • Weakness based upon angular size, close prey may
    be interpreted as predator. Therefore the
    threshold is essential (arbitrary as no data
    exists on predation)

  • No additional neurons, based upon divergent
    patterns of tectal neurons projections
  • Consistent with biological lesion data
  • codes for closeness
  • realistic precision (about 30)
  • Depends on the movement direction

Reproduced Phenomena
  • Lesion and stimulation experiments
  • Lesion and stimulation of the optic tectum
  • Lesion of the pretectum
  • Generation of saccadic movements
  • Monocularized salamanders
  • Prey preferences

Saccadic Movements
  • Pursuit movements such as the head accelerates
    for a few seconds, until maximum velocity is
    reached, and then is released
  • We attribute them to the tectal cells resolution

Monocularized Salamander
  • With one eye covered, H.Italicus shows a
    conspicuous approach behavior toward a prey
    stimulus. It takes a curved path and bends its
    body toward the side of a seeing eye,
    compensating by turning the head between 60 and

Monocularized Salamander
Prey preferences
  • All preferences are inherent to the network!!

Comparison to Previous Works
  • Simulander I
  • More neurons, but still biologically plausible
    (2000 vs. 100)
  • Less accurate, more realistic (2-6 vs. 1)
  • Inherent preferences vs. a function reflecting
    the stimulus size and velocity (corresponds to
    the observers knowledge)
  • No real distribution of tectal neurons, respected
    in Simulander
  • Faster reaction
  • No positions in which stationery prey elicit
    orienting behavior
  • Simulander II
  • Lower precision, but more realistic (90, real
    success rate 40)
  • In Simulander far objects elicit more activity,
    double inconsistency (should code for closeness
    and further objects seem smaller)

Response to questions
  • Extraction of stimulus localization and depth
    estimates can be achieved with a structured
    mapping between the optic tectum and the brain
  • The sensorimotor transformation of the horizontal
    angular distance of the tectum neurons to muscle
    activity can provide an accurate prey
  • Direct observation of the influence of head
    motion during the approach is provided.
    Additional mechanisms for dealing with the
    self-motion visual shifts are necessary
  • The investigated tectum model accounts for the
    typical curved approach in monocularized
  • A plausible mechanism that acts as a releaser for
    the snapping behavior is proposed

  • We have implemented models of the three types of
    retinal ganglion cells, the optic tectum, the
    pretectum and a tongue projection mechanism that
    account for the typical feeding sequence and
    escape behavior
  • The optic tectum model reproduces many
    experimental data
  • Everything is observable
  • Warning data and methodology dependant
  • Our salamander resembles a newly born salamander
    thrown in the world

Future Work
  • Study a tectum model with nonlinear weights
  • Use the real distribution and receptive fields
    sizes of tectal neurons
  • Time-dynamics vs. discrete time steps
  • Study the effect of overlapping fields
    (redundancy gt error resistant, maybe emerging
  • Implement a visual attention model
  • Implement experience-based models such as
  • Further development of the pretectum model
  • Extend the model to other brain areas such as the
    nucleus isthmi or thalamus (obstacle avoidance)
  • Develop a more elaborate model for depth
    estimation (not only frontal)
  • Work on an object-background discrimination with
    respect to self-motion shifts of the visual input