From Neuroscience to Mechatronics

1
From Neuroscience to Mechatronics

• Presentation by Fabian Diewald
• JASS April 2006

2
Outline

• The human brain
• The vestibulo-ocular reflex (VOR)
• The VOR technical applications
• Camera stabilizing system at TU Munich
• A possible algorithm
• Testing the camera stabilizing system
• Another field of neuroscience and technical
  application face recognition
• Conclusion and outlook

3
Learning from nature a justified strategy
Consequently nature has a great time advantage!
4
The human brain

• cerebellum "little brain"
• responsible for accurate movement
• instructions by the forebrain insufficient
• instructions have to be translated into accurate
  commands by the cerebellum
• cerebellum essential part in learning motor skills

5
Neurons elementary components of the central
nerve system

• dendrites "input" of a neuron
• axon "output"
• axon terminals/boutons contact other neurons or
  muscles

6
Neurons elementary components of the central
nerve system

• transmission works electrically
• information through the axon is encoded in
  changes of electrical potential
• short impulses with fixed intensity
• information is contained in firing
  frequency("pulse rate modulation")

7
Synapses link between neurons

• gap against electrical transmission
• transmission between neurons is controlled by
  chemicals called neurotransmitters
• controlled transmission is important in regard to
  adaptation and learning

8
The structure of the cerebellar cortex
parallel fibresaxons of granule cells,synapse
on Purkinje cells
Purkinje cells 200 000 synapses per cell, only
output for cerebellar cortex
granule cells, 20 per mossy fibre, half of all
neurons are granule cells
mossy fibres information input, synapse on
granule cells
9
The influential theory of Marr-Albus

• twice independently proposed
• Marr, David A Theory of Cerebellar Cortex,
  1969, Journal of Physiology 202 437-470
• Albus, James S. "A theory of cerebellar
  function", 1971, Mathematical Biosciences 10
  25-61
• major aspects
• climbing fibre carries a teaching signal
• signal influences the synapses of Purkinje cells
• consequently the intensity of certain inputs to
  the Purkinje cells can be controlled

10
The influential theory of Marr-Albus
disappointment by Marr himself

• Marr 1982"In my own case, the cerebellar study
  (...) disappointed me, because even if the theory
  was correct, it did not enlighten one about the
  motor system it did not, for example, tell one
  how to go about programming a mechanical arm."

11
Outline

• The human brain
• The vestibulo-ocular reflex (VOR)
• The VOR technical applications
• Camera stabilizing system at TU Munich
• A possible algorithm
• Testing the camera stabilizing system
• Another field of neuroscience and technical
  application face recognition
• Conclusion and outlook

12
The vestibulo-ocular reflex (VOR)

• aim stable picture of our environment/object we
  are staring at in spite of moving head
• only use of visual information would be too slow
  for stabilization (visual processing delay about
  50-100ms)
• signals of the vestibular organ (motion of the
  head) are more or less directly transmitted to
  the extra-ocular eye muscles, leading to process
  times of 5-10ms
• vestibular signals are also influenced by visual
  information

13
The vestibulo-ocular reflex (VOR)
experimentation with monkeys

• VOR does an excellent job in monkeys as well as
  in other vertebrates under normal conditions
• magnifying or miniaturizing glasses cause
  abnormal image motion speed perceived during head
  motion
• monkeys with glasses show problems with head
  motion in the beginning
• after several days considerable improvement is
  completed
• then taking the glasses away
• monkeys show problems again
• after some days normal behaviour again
• unconscious use of VOR, motoric issue,
  calibration needed
• ? typical case concerning the cerebellum

14
The vestibulo-ocular reflex paths of
information
eye (visual information)
vestibular organ (head motion)

• adaptive path
• over Purkinje cells
• transferred on parallel fibres

• direct path
• very quick
• not accurate (not sensible to changes in
  the system, e.g. changes in eye muscle
  strength)

inferior olive

• adaptation path
• teaching signal
• transferred on climbing fibres
• "strengthens" or "weakens" the synapses

synapses ofPurkinje cells
Purkinje cells
cerebellum
eye muscles
15
Outline

• The human brain
• The vestibulo-ocular reflex (VOR)
• The VOR technical applications
• Camera stabilizing system at TU Munich
• A possible algorithm
• Testing the camera stabilizing system
• Another field of neuroscience and technical
  application face recognition
• Conclusion and outlook

16
The vestibulo-ocular reflex possible technical
applications
17
The vestibulo-ocular reflex also an important
automotive application

• wide angle cameras used to obtain an overview of
  the environment
• for a closer look (road signs, number plates
  etc.) a telephoto lens is needed, which is quite
  sensitive to motion

18
Requirements concerning driver-assistance systems

• camera stabilization only by optical means is too
  slow
• inertial measurement of head angular velocity
  needed
• affordable hardware
• cheap inaccurate sensors must be allowed
  (multi-sensor fusion of translation/angular
  velocity and visual information)
• inaccuracies have to be compensated
  automatically
• ? similar problems/requirements as in biology
• the vestibulo-ocular reflex may solve our
  technical problem!

19
Outline

• The human brain
• The vestibulo-ocular reflex (VOR)
• The VOR technical applications
• Camera stabilizing system at TU Munich
• A possible algorithm
• Testing the camera stabilizing system
• Another field of neuroscience and technical
  application face recognition
• Conclusion and outlook

20
The VOR biological and technical analogons
How can we implement cerebellar structures as
computer hardware?

• the structure of the cerebellum is quite
  homogeneous although different information
  appears in different regions, these regions are
  similar
• idea of "cerebellar chip"
• today normally no need for cerebellar chip as
  DSPs are very multifunctional and cerebellar
  algorithms can be implemented software-based
  (e.g. in Matlab)

21
The VOR biological and technical analogons
mechatronics
neuroscience/biology

• vestibular organ in the inner ear 6-DOF inertial
  measurement unit
• direct pathway vestibular to muscles CAN bus
  system
• cerebellum DSP with specific algorithm
• extra-ocular eye muscles servo motors

main difficulties

• commercially available inertial measurement units
  are too big for driver-assistance systems
• the algorithm biology uses in the cerebellum has
  to be detected and implemented, which is
  obviously difficult(remember Marr's quotation!)

22
Camera stabilization system at TU Munich,
Institute of Applied Mechanics

• serial gimballed configuration
• two perpendicular axes
• pan actuator driven by4.5 Watt motor
• inner frame driven by11 Watt Maxon motor
  withbacklash free HarmonicDrivegearbox
• each axis controlled by differential encoders
  with 512 steps
• camera CMOS sensor with effective resolution of
  750x400 pixels
• in the future only a mirror is moved leading to
  further miniaturization

23
Coping with difficulty 1size of inertial
measurement unit (IMU)

• smallest available intelligent IMU 50x38x25 mm³
• too big for driver-assistance-systems
• new IMU was developed within the FORBIAS project

edge length 15 mm
3 gyroscopes(rotation), bandwith up to 100Hz
accelerometer with 3 axes on one
chip(translation), bandwith up to 640Hz
24
Outline

• The human brain
• The vestibulo-ocular reflex (VOR)
• The VOR technical applications
• Camera stabilizing system at TU Munich
• A possible algorithm
• Testing the camera stabilizing system
• Another field of neuroscience and technical
  application face recognition
• Conclusion and outlook

25
Coping with difficulty 2 (main difficulty)implem
enting an adequate algorithm (1)

• central ideas/principles
• motor movements are calculated according to
  angular velocity of measurement unit ("direct
  path")
• theoretically (perfect sensors/perfect
  production) this is sufficient for camera
  stabilization
• but due to inaccuracies in production and
  sensors themselves there is always relative
  movement of the picture of the environment
  ("optical flow")
• the relation between angular rates and this
  relative movement is calculated
• the result dynamically influences a matrix w

26
Coping with difficulty 2 (main difficulty)implem
enting an adequate algorithm (2)

• central ideas/principles
• this matrix w calculates a certain output out of
  angular velocity
• this output is used to "clean" the signal of the
  direct path (velocity) additively
• result after some time the matrix w is "perfect"
  (learning completed) and always knows what to add
  to the measured angular velocity so that there is
  no optical flow any more, i.e. optical flow and
  angular velocity are decorrelated
• consequently further cost-reduction possible by
  teaching the system after assembling and storing
  the matrix w on a chip
• but in this case inaccuracies because of e.g.
  plastic mechanical deformation during use cannot
  be compensated

27
The vestibulo-ocular reflex paths of
information
eye (visual information)
vestibular organ (head motion)

• adaptive path
• over Purkinje cells
• transferred on parallel fibres

• direct path
• very quick
• not accurate (not sensible to changes in
  the system, e.g. changes in eye muscle
  strength)

inferior olive

• adaptation path
• teaching signal
• transferred on climbing fibres
• "strengthens" or "weakens" the synapses

synapses ofPurkinje cells
Purkinje cells
cerebellum
eye muscles
28
Coping with difficulty 2 (main difficulty)implem
enting an adequate algorithm
central idea decorrelation of angular rates and
optical flow, decorrelator contains dynamic
state matrix w, dynamically influenced by optical
flow, angular velocity and its derivative
(optical flow teaching signal)
Decorrelator with weight matrix
,equivalent to synapses of Purkinje cells
Ddecomposition to "parallel fibre signals"
Sdiagonal sensitivity matrix
brainstem B direct path
I
P
C
w


-
from Günthner, W. Glasauer, S. Wagner, P.
Ulbrich, H. Biologically inspired multi-sensor
fusion for adaptive camera stabilization in
driver-assistance systems, Advanced Microsystems
for Automotive Applications AMAA, Berlin, April
25-27, 2006 (in press)
29
Coping with difficulty 2 (main difficulty)implem
enting an adequate algorithm
central idea decorrelation of angular rates and
optical flow, decorrelator contains dynamic
state matrix w, dynamically influenced by optical
flow, angular velocity and its derivative
(optical flow teaching signal)
Decorrelator with weight matrix
,equivalent to synapses of Purkinje cells
Ddecomposition to "parallel fibre signals"
Sdiagonal sensitivity matrix
brainstem B direct path
I
P
C
w


-
from Günthner, W. Glasauer, S. Wagner, P.
Ulbrich, H. Biologically inspired multi-sensor
fusion for adaptive camera stabilization in
driver-assistance systems, Advanced Microsystems
for Automotive Applications AMAA, Berlin, April
25-27, 2006 (in press)
30
Coping with difficulty 2 (main difficulty)implem
enting an adequate algorithm
central idea decorrelation of angular rates and
optical flow, decorrelator contains dynamic
state matrix w, dynamically influenced by optical
flow, angular velocity and its derivative
(optical flow teaching signal)
Decorrelator with weight matrix
,equivalent to synapses of Purkinje cells
Ddecomposition to "parallel fibre signals"
Sdiagonal sensitivity matrix
brainstem B direct path
I
P
C
w


-
from Günthner, W. Glasauer, S. Wagner, P.
Ulbrich, H. Biologically inspired multi-sensor
fusion for adaptive camera stabilization in
driver-assistance systems, Advanced Microsystems
for Automotive Applications AMAA, Berlin, April
25-27, 2006 (in press)
31
Possible variants of the basic systems (1)
central idea decorrelation of angular rates and
optical flow, decorrelator contains dynamic
state matrix w, dynamically influenced by optical
flow, angular velocity and its derivative
(optical flow teaching signal)
Decorrelator with weight matrix
,equivalent to synapses of Purkinje cells
Ddecomposition to "parallel fibre signals"
Sdiagonal sensitivity matrix
brainstem B direct path
I
P
C
w


-
from Günthner, W. Glasauer, S. Wagner, P.
Ulbrich, H. Biologically inspired multi-sensor
fusion for adaptive camera stabilization in
driver-assistance systems, Advanced Microsystems
for Automotive Applications AMAA, Berlin, April
25-27, 2006 (in press)
32
Possible variants of the basic systems (2)
central idea decorrelation of angular rates and
optical flow, decorrelator contains dynamic
state matrix w, dynamically influenced by optical
flow, angular velocity and its derivative
(optical flow teaching signal)
Decorrelator with weight matrix
,equivalent to synapses of Purkinje cells
Ddecomposition to "parallel fibre signals"
Sdiagonal sensitivity matrix
brainstem B direct path
I
P
C
w


-
from Günthner, W. Glasauer, S. Wagner, P.
Ulbrich, H. Biologically inspired multi-sensor
fusion for adaptive camera stabilization in
driver-assistance systems, Advanced Microsystems
for Automotive Applications AMAA, Berlin, April
25-27, 2006 (in press)
33
Outline

• The human brain
• The vestibulo-ocular reflex (VOR)
• The VOR technical applications
• Camera stabilizing system at TU Munich
• A possible algorithm
• Testing the camera stabilizing system
• Another field of neuroscience and technical
  application face recognition
• Conclusion and outlook

34
Other visual technical problems to be solved for
driver assistance

• optical flow caused by camera rotation can
  successfully be removed by camera stabilization
  as just seen, but...
• line of sight may have to be kept on the road
• rapid changes of viewing direction ("saccades")
  have to be implemented
• furthermore closer examination of certain
  objects (signs etc.) need visual tracking

35
Testing of the camera stabilization system under
laboratory conditions

• testing in the laboratory by mounting the system
  on a hexapod
• created sensed angular rates of 100/s
• optical flow was reduced from 6 pix/frame to less
  than 1 pix/frame after 2-3 minutes of adaptation
• the improvement within this time was also
  subjectively viewable

36
Testing of the camera stabilization system "on
the road" (1)

• system mounted near the rear-view mirror
• additional camera installed as well to detect
  points of interest for the camera with telephoto
  lens
• system tested in association with saccade and
  visual tracking
• vehicle velocity and yaw rate added to the system
  via CAN bus to improve tracking of space fixed
  objects

37
Testing of the camera stabilization system "on
the road" (2)

• three modes were tested
• lane marker was focused in a distance of 60m,
  followed up to a distance of 20m, then the next
  was focused on
• lane separation was focused on in a constant
  distance of 40m
• nothing was focused on but the camera was
  stabilized around a constant line of sight
• in all modes bumps could be compensated

38
Outline

• The human brain
• The vestibulo-ocular reflex (VOR)
• The VOR technical applications
• Camera stabilizing system at TU Munich
• A possible algorithm
• Testing the camera stabilizing system
• Another field of neuroscience and technical
  application face recognition
• Conclusion and outlook

39
Learning from the human brain another example

• a current example face recognition
• excellent recognition abilities by humans under
  different circumstances, e.g.
• illumination
• viewing angle
• pose
• facial expression
• ...
• need for technical face recognition systems
  (e.g. war against terrorism)
• consequently need to continue exploring the
  neuroscience of face recognition

40
Face recognition the problem with different
angles of view

• How can we/a computer tell that each face
  belongs to the same person?

41
Face recognition adaption of weights and output
as weighted sum
testing step unfamiliar view of the face, but
nevertheless the highest output is produced for
"Betty"
"hidden units" one for each angle of view the
more similar the input is to the "prototype" face
of one unit, the more "active" the unit gets
learning step changing the weights within the
network so that the output is "1" for the viewed
person
from Vatentin, D. Abdi, H.. Edelman, B. What
represents a face A Computational Approach for
the Integration of Physiological and
Psychiological Data, 1997
42
Outline

• The human brain
• The vestibulo-ocular reflex (VOR)
• The VOR technical applications
• Camera stabilizing system at TU Munich
• A possible algorithm
• Testing the camera stabilizing system
• Another field of neuroscience and technical
  application face recognition
• Conclusion and outlook

43
Conclusion and outlook

• technical use of neuroscience is the key to
• giving machines several typical human abilities
• optimizing service intervals and
• minimizing complexity of installing systems by
  self-learning abilities
• cost reduction
• ...
• consequently interesting to a large variety of
  technical fields

44
Thank you!

• Thank you for your attention!
• Fabian DiewaldFabian.Diewald_at_mytum.de