Lecture 7: Basics of Neural Nets and Past-Tense model

About This Presentation

Title:

Lecture 7: Basics of Neural Nets and Past-Tense model

Description:

COM1070: Introduction to Artificial Intelligence: week 10 Yorick Wilks Computer Science Department University of Sheffield www.dcs.shef.ac.uk/-yorick – PowerPoint PPT presentation

Number of Views:126

Avg rating:3.0/5.0

Slides: 58

Provided by: Lucy142

Category:

more less

Transcript and Presenter's Notes

Title: Lecture 7: Basics of Neural Nets and Past-Tense model

1
COM1070 Introduction to Artificial Intelligence
week 10 Yorick Wilks Computer Science
Department University of Sheffield www.dcs.shef.ac
.uk/-yorick
2
Characteristics of von Neumann architecture

A von Neumann machine is a sequential processor
executes instructions in a program one after
another.
Each string of symbols stored at specific memory
location.
Access to stored string via numerical address of
strings location.
Single seat of control, central processing unit,
CPU.
But brain may still be a computer, just not a von
Neumann machine, or possibly a VNM with enormous
redundancy of data.

3
Ability of GOFAI and Neural Nets to provide an
account of thought and cognition

The debate is about whether it is possible to
provide a neural computing (NN) account of
cognition, or whether we should assume that a
symbol system (GOFAI, or Good Old Fashioned AI)
is required.
Physical symbol system hypothesis it is possible
to construct a universal symbol system that
thinks
Strong symbol system hypothesis only universal
symbol systems are capable of thinking.
I.e. anything which thinks (e.g. human brain)
will be a universal symbol system.
I.e. all human thinking consists of symbol
manipulation.
I.e. only computers can think therefore we are
computers.
If its not a universal symbol system, it cant
think!

Pylyshyn an advocate of strong symbol system
hypothesis
Our mental activity consists of manipulation of
sentence like symbolic expressions.
Human thought manipulation of sentences of
internal mental language.
Fodors Language of Thought or Mentalese
We can show that brain does not have a von
Neumann architecture.
But this does not disprove Strong Symbol System
Hypothesis.
Strong symbol system hypothesis says nothing
about architecture.

Could have symbol manipulation in parallel system
Could have different memory system i.e.
content-addressable storage
Instead of CPU could have different parallel
control.
Brain can form representations of events in
world.
SSSH representations in brain take form of
sentence-like strings of symbols.
I.e. sentence Fido licks Brian is symbolic
representation of fact that Fido licks Brian.
Language of thought, or Mentalese
Mentalese representations not literally English,
but like sentences, in that they have basic
vocabulary units, and form in which they are
arranged determines their meaning.

Advantages of mentalese
Provides an account of beliefs, intentions and
doubts etc. these are also expressed in terms of
mentalese sentences.
Provides an account of productivity and
systematicity.
Human language is productive, (no limit to number
of sentences we can produce).
And systematic (if you can say John loves Mary,
you can also say Mary loves John)
Human thought productive and systematic because
it relies on Mentalese, which is productive and
systematic.
Origin of all this in Chomskys generative
linguistic theories.

Neural Computing can it form an alternative to
SSSH?
Is it possible to provide a connectionist account
of thought, or is it as SSSH advocates would
claim, impossible?
Symbols Subsymbolic Hypothesis versus Strong
Symbol system hypothesis
Symbol System Hypothesis (Newell and Simon, 1976)
..a physical symbol system has the necessary and
sufficient means for general intelligent
action..
A symbol designates something.
A symbol is atomic (cannot be broken down
further).
E.g. elephant designates an elephant. Or P
could designate elephant. Or 01100 could
designate elephant, but with no interpretation of
the 1s and 0s.

Compositional symbol compound symbol which has
meaningful parts, whose overall meaning is
determined by meaning of those parts.
E.g. sentences of natural language.
The kangaroo jumped over the elephant.
Distinction between symbol types and symbol
tokens
E.g. in AMBIGUITY, 9 letter tokens, 8 letter
types.
Same symbol type can be realised in many
different ways.
Symbol is both (a) a computational token, and (b)
it designates something, i.e. it is a
representation.

9
Connectionism subsymbolic hypothesis

(see Smolensky, 1988).
Symbols can be broken down, computation takes
place at subsymbolic level.
Connectionist representations distributed
represent-ation.
Distributed representation pattern of activity
over several nodes.
E.g. in a distributed representation of
elephant, elephant is represented by
distributed represent-ations over several nodes.
Thus, there is not an atomic symbol for elephant.

So connectionism rejects Strong Symbol System
Hypothesis
To complicate matters, one can have localist
neural networks, where concepts are represented
by single nodes. (e.g. node which represents
elephant). Localist connectionism assumes its
symbols given and may be compatible with the
SSSH.
This discussion applies to distributed
(non-localist) representations.

Distributed representations subsymbolic
computation.
E.g. representing letter A
Localist scheme, single unit for A, one for B
etc.
Or (in a subsymbolic system) letters represented
as patterns of activity across 78 units. E.g. A
units 1,2, and 3, B units 4,5, and 6 etc.
Individual units stand for features of letters.
Thus letter A will be joint activity of various
features it contains. So letter E will share
several features with letter F. Thus similarities
and differences among items are reflected in
similarities and differences among
representations.

Symbolic-Subsymbolic distinction.
In a symbolic system the computational level
coincides with the representational level. In a
subsymbolic system, the computational level lies
beneath the representational level.
Symbol is both (a) a computational token, and (b)
it designates something, i.e. it is a
representation.
But in subsymbolic Connectionism representations
across several units, but computational tokens
are units.
But does the same issue reappear? Subsymbolic
computation may not assume the symbols but what
about their features--e.g. horizontal strokes for
E and F??

13
Symbolic criticisms of connectionism

Arguments in favour of Strong Symbol System
Hypothesis
Fodor and Pylyshyn (1988)
Argument that connectionism is inadequate as a
representational system.
Focus on issues of compositionality and
systematicity
Fodor and Pylyshyn (1988) argue
Compositionality and structure sensitivity are
necessary to support cognitive processes.
Only classical representations exhibit
composition-ality and structure sensitivity
(systematicity)
Only classical representations can support
cognitive processes.

Compositionally concatenative representations
Where complex representations are composed from
primitive tokens combined concatenatively
Molecular representations can be formed out of
constituents, and can be manipulated in accord
with syntactic rules.
E.g. the kangaroo jumped over the elephant
If you can understand this, you can also
understand a sentence which has the same words in
a different order. (Systematicity).
E.g. The elephant was jumped over by the
kangaroo
or
The elephant jumped over the kangaroo

Two sentences are composed of same elements.
Constituent elements manipulated according to
syntactic rules.
But, according to Fodor and Pylyshyn,
connectionist representations cannot be
manipulated like this.
FP cannot compose simple connectionist
representations into more complex representations
Much of this argument comes down to the role of
tree structures or hierarchies, which are needed
to express syntactic relationships --- SSSH
people say trees cannot be learned by
connectionist systems..
But Jordan Pollack showed in 1986 that they can
(up to a point anyway)--this argument is very
like a return of the XOR argument at a higher
level.

16
Connectionist Counter-Arguments

Distributed representations nonsymbolic and
continuous (developed over hidden units).
But van Gelder (1990), can have functional
compositionality
.. We have functional compositionality when
there are general, effective and reliable
processes for (a) producing an expression given
its constituents, and (b) decomposing the
expression back into those constituents
Many nets can do (a) and (b)
For example, net can be trained to structurally
disambiguate sentences.
John saw money with the telescope.
(John (saw money) (with the telescope)).

Complex representations of input sentences
developed over hidden units.
Then these representations decomposed into
required output (ii).
Thus, fully distributed representations carry
information about syntactic structure of inputs,
without being syntactically structured.
I.e. they demonstrate a functional
compositionality
by moving from structures to their components and
back again.

.
But can distributed representations permit
structure sensitive operations? (systematicity)
(e.g. changing from active to passive).
Fodor and McLaughlin (1990) to support
structure sensitive operations, representations
must contain explicit tokens of original
constituent parts of complex expression.
But can show that connectionist representations
can have property of systematicity (permitting
structure sensitive operations).

Chalmers (1990), trained a connectionist net to
transform representations of active sentences, to
passive sentences.
Developed representations for both active and
passive sentences. Used RAAM nets (Pollack, 1990)
to do this.
Took the fully distributed representations from
the RAAM nets, for both active and passive
sentences.
Trained a net to translate from active sentences
to passive sentences.
Training complete could input active sentences,
extract representation, and translate that into
the representation for a passive sentence, which
could be decoded into the passive sentence.

Also generalised to sentences that were not part
of the training set.
I.e. Connectionist representations did permit
structure sensitive operations, without being
decoded into symbols.

21
Summary of representation argument

Claimed by Fodor and Pylyshyn that
Compositionality and structure sensitivity are
necessary to support cognitive processes.
Only classical representations exhibit
composition-ality and structure sensitivity
(systematicity)
Only classical representations can support
cognitive processes.
BUT
Can demonstrate that connectionist nets show
functional compositionality. Although
representation do not contain tokens, as symbolic
representations do, these tokens can be obtained
from the representations.

E.g. Can train nets to disambiguate sentences
like the following. Intervening hidden unit
representation does not contain explicit tokens,
but output does.
John saw money with the telescope.
(John (saw money) (with the telescope))
Can also show that connectionist representations
can support structure sensitive operations
(systematicity).
Chalmers (1990) Translating from active to
passive sentences using connectionist distributed
representations.
So counter-argument against claims that neural
nets cannot provide an account of cognition. So
Neural Computing can provide an alternative to
the Strong Symbol System Hypothesis.

But has yet to be shown if can provide a
connectionist account of all aspects of thought.
Smolensky, (1988) .. It is far from clear
whether connectionist models have adequate
power to perform high level cognitive tasks..
Connectionism Does well at accounting for
low-level aspects of cognition e.g. movement,
pattern recognition.
But Beliefs? Intentions?
Can Neural Computing provide an account of
Direct Conscious Control? (Norman, 1986)
E.g. consciously planning what to do,
introspecting about our thoughts, holding
beliefs, making logical inferences, early stages
of learning a skill.

Possible that brain best modelled in terms of
hybrid system
Connectionist account of lower level processes
Symbolic account of higher level processes.

25
Adaptive Behaviour, and Symbol Grounding
Revisited

Three different approaches to Artificial
Intelligence, two of which we have already
encountered many times
Symbolic AI or Traditional AI or GOFAI (Good Old
Fashioned AI)
Neural Computing or Connectionism or Parallel
Distributed Processing
Adaptive Behaviour or Behaviour-based Robotics
Adaptive Behaviour
Rodney Brooks, at MIT Artificial Intelligence
Lab. (see reference to Brooks on web page for the
course)
Brooks, R.A. (1991) Intelligence without Reason.
MIT AI Lab Memo 1293, April 1991

An example Allen, a reactive robot. (named after
Allen Newell?)
Sonar sensors, and odometer to keep track of
distance travelled.
Controlled by cable from off-board special
purpose computer.
Lowest level reactive layer used sonar readings
to keep away from moving and static obstacles. -
if an obstacle is close, instead of bumping into
it, stop.
Second level random wandering. Every 10 seconds,
generate a movement in a random direction.
Third level Look for a distant place, and move
towards it. Odometry can be used to monitor
progress.
Three layers made it possible for robot to
approach goal, whilst avoiding obstacles.

Goal subsumption switching control between the
modules is driven by the environment, not by a
central locus of control.
Robot heads for goal until sensors pick up
information that there is an obstacle in the way.
The obstacle avoidance module cuts in. Once the
obstacle has been avoided the goal-finding module
takes control again.
Robot can move around in the environment although
it does not build, or use, any map of that
environment, and is only driven by simple
environmental cues.

Second example Herbert (Herbert Simon?)
Wanders about an office environment, picking up
coke cans and returning them to start point.
Sensors infrared ports, and laser 3D data.
Actuators motors driving wheels, and manipulator
arm with sensors.
Subsumption architecture several
behaviour-generating modules.
Modules include obstacle avoidance, wall
following, and recognition of coke cans.
Control of modules Only suppression and
inhibition between alternative modules - no other
internal communication.
Each module connected to sensors and to
arbitration network which decides which competing
action to take.

Description of Herbert in action
When following a wall, Herbert spots a coke can.
The robot locates itself in front of the can.
The arm motion is then begun - when can is
detected with sensors local to the arm, it is
picked up.
Advantages naturally opportunistic. If coke can
put right in front of Herbert, can collect it and
return to start, since no expectations about
where coke cans will be found. Can find coke
cans in a variety of locations, even if never
found there before.
But.

30
(No Transcript)
31
Behaviour-based Robotics

Idea of building autonomous mobile robots.
New approach, where robots operate in the world,
and use highly reactive architectures, with no
reasoning systems, no manipulable
representations, no symbols, and totally
decentralized computation (Brooks, 1991)
I wish to build completely autonomous mobile
agents that co-exist in the world with humans,
and are seen by those humans as intelligent
beings in their own right. I will call such
agents Creatures... (Brooks, 1991)
Brooks, R. (1991) Intelligence without
Representation Artificial Intelligence, 47,
139-159.

See Elephants dont play chess, (1990 paper by
Brooks)
Brooks, Rodney A. (1990) Elephants dont play
chess. In Pattie Maes (Ed) Designing autonomous
Agents, Cambridge, Mass MIT Press.
Because elephants dont play chess, no reason to
assume they are not intelligent.
Emphasis on kind of behaviour exemplified by
elephants, rather than on more abstract human
behaviours (e.g. games, speech recognition,
problem solving).

A Creature must cope appropriately and in a
timely fashion with changes in its dynamic
environment.
A Creature should be robust with respect to its
environment minor changes in the properties of
the world should not lead to total collapse of
the Creatures behaviour rather one should only
expect a gradual change in the capabilities of
the Creature as the environment changes more and
more.
A Creature should be able to maintain multiple
goals and, depending on the circumstances it
finds itself in, change which particular goals it
is actively pursuing thus it can both adapt to
surroundings and capitalize on fortuitous
circumstances.
A Creature should do something in the world it
should have some purpose in being.

Set of principles (Brooks, 1991)
The goal is to study complete integrated
intelligent autonomous agents.
The agents should be embodied as mobile robots
situated in unmodified worlds found round
laboratory. (embodiment).
Robots should operate under different
environmental conditions - e.g. in different
lighting conditions, when sensors and actuators
drift in calibration (situatedness).
Robots should operate on timescales commensurate
with timescales used by humans (situatedness).

35
Key Topics of Behaviour-based Approach

Situatedness
Embodiment
(animal or insect) Intelligence
Emergence

36
Situatedness

A situated automation is a finite-state machine
whose inputs are provided by sensors connected to
the environment, and whose outputs are connected
to effectors.
The world is its own best model
Traditional AI, working in symbolic abstracted
domain.
Problem solvers which are not participating in
the world as agents.
Dealing with model world - no real connection to
external world.

Alternative approach, to use a mobile robot which
uses the world as its own model, referring to
information from sensors rather than internal
world model.
Representations are developed which capture
relationships of entities to robot.
Situated agent must respond in timely fashion to
inputs but much information from the world.

38
Embodiment

The world grounds symbolic regress
Embodiment Physical grounding of robot in real
world.
According to Brooks (1991), embodiment is
critical for 2 reasons.
Only an embodied agent is validated as one that
can deal with real world.
Only through a physical grounding can any
internal symbolic system be given meaning.

39
Brooksian view of Intelligence

Intelligence is determined by the dynamics of
interaction with the world
Some activities we think of as intelligent have
only been taking place for a small fraction of
our evolutionary lineage.
Simple behaviours to do with perception and
mobility took much longer to evolve.
Would make sense to begin by looking at simpler
animals.
- looking at dynamics of interaction of robot
with its environment.

40
Emergence

Intelligence is in the eye of the observer
Intelligence emerges from interaction of
components of the system.
Behaviour-based approach - intelligence emerges
from interaction of simple modules.
e.g. Obstacle avoidance, goal finding, wall
following modules.

Main ideas
No central model maintained of world
No central locus of control
No separation into perceptual system, central
system and actuation system
Behavioural competence improved by adding one
more behaviour specific network to existing
network. Crude analogy to evolutionary
development
No hierarchical development
Layers or behaviours run in parallel

42
Criticisms?

This approach wont necessarily lead to system
capable of more complex behaviours. A new
controller is needed for each task.
The experimenter is deciding on what modules to
add, and what environment and task the robot
should be exposed to. - not the same as
evolution.
But in terms of evolution, new behaviours and new
mental structures are learnt in response to the
environment, not added by an experimenter.
Similarly, in the development of an individual,
new representational structures are developed in
response to the environment, not added by an
experimenter.

It would be more impressive if the robot learnt
new behaviour modules in response to the
environment. This possibility is discussed by
Brooks (1991), but has not yet been successfully
tackled.
Emphasis in this approach on reacting to the
environment. And it is the case that apparently
quite sophisticated behaviours can result from
simple reaction to the environment. But
representations are needed for more complex
tasks.
e.g. Find an empty can and bring it back to the
starting point
requires the formation of an internal
representation corresponding to a map. Need to
provide an account of the development of
representations.

44
Symbol Grounding revisited

Traditional view the language of thought (Fodor,
1975), that The mind is a symbol system and
cognition is symbol manipulation.
Advocates of symbolic model of mind (e.g. Fodor,
and Pylysyn) argue that symbol strings capture
what mental phenomena such as thoughts and
beliefs are.
Symbol system symbols (arbitrary physical
tokens) manipulated on the basis of explicit
rules.
Rule-governed symbol manipulation is based on
syntax of symbol tokens (not their meaning).

Symbols can be rulefully combined primitive
atomic symbol tokens can be combined to form
composite symbol-token strings.
Resulting symbol-token strings can be given a
meaning - i.e. they are semantically
interpretable.
BUT approach of assuming that mind is symbol
system can be criticised - in terms of symbol
grounding.

A criticism of symbol systems is that symbol
system capable of passing the Turing Test will
not be a mind, because the symbols have no
semantics (meaning) (remember the Chinese Room)
From Searle (1997) The Mystery of Consciousness
Programs are entirely syntactical
Minds have a semantics
Syntax is not the same as, not be itself
sufficient for, semantics
Therefore programs are not minds. QED

It does not matter how well the system can
imitate the behaviour of someone who really does
understand, nor how complex the symbol
manipulations are you cannot milk semantics out
of syntactical processes alone (Searle, 1997).

48
Symbol grounding, as discussed by Stevan Harnard

Harnard, S (1990) The Symbol Grounding Problem.
Physical D 42, 335-346.
Copy of paper can be obtained from
http//www.cogsci.soton.ac.uk/harnad/genpub.html
N.B. see the relevant websites for this course at
http//www.dcs.shef.ac.uk/yorick/ai_course/ai-cou
rse.html

Computation consists of manipulation of
meaningless symbols.
For them to have meaning they must be grounded in
non-symbolic base.
Like the idea of trying to learn Chinese from a
Chinese dictionary.
Standard reply of symbolist (e.g. Fodor, 1980) is
that the meaning of the symbols comes from
connecting the symbol system to the world in the
right way.
But how could this be done?
Harnard provides one possible solutionSymbols
need to have some intrinsic semantics or real
meaning.

For Harnard, symbols are grounded in iconic
representations of the world.
e.g. consider the symbol horse
iconic representation of a horse, is a
representation of the shapes that horses cast on
our retinas (i.e. sensory surface of the eye).
From these iconic representations (many from
individual views of horses), we form a
categorical representation - that captures the
features we need to identify a horse.
Thus the name horse is grounded in iconic and
categorical representations, learned from
experience.

Similarly, stripes is grounded in iconic and
categorical representations, learned from
experience.
These symbols can be combined
zebra horse stripes.
New symbol of zebra, is built up from the
grounded representations of horse and
stripes, which gives it meaning.
Harnard is proposing a hybrid system, in which
thought is assumed to be symbol manipulation, but
the symbols are grounded in iconic and
categorical representations of the world.
The problem with all this is WHICH symbols are so
grounded (peace, courage, Hamlet?)

Other solutions to symbol grounding problem have
been proposed.
Essential idea is that symbols need to be given
some meaning - need for grounding in meaningful
representations, to escape from circularity of
defining symbols in terms of symbols .
2 other (partial) solutions
Adaptive Behaviour and embodiment
Connectionist (neural computing)
Both are robot-prosthetic arguments, the first
without representations and the second with
implicit ones.

53
Symbol Grounding and Adaptive Behaviour

Would make sense to have symbols physically
grounded in real world.
Embodiment
Only an embodied agent is validated as one that
can deal with real world
Only through a physical grounding can any
internal symbolic system be given meaning.
But adaptive behaviour people dont want to have
symbols, grounded or not.

Suggests a new approach to grounding symbolic
representations - but as yet no clear account of
how symbols might emerge due to such interactions
with the real world.
Emphasis in work on behaviour-based robotics has
been on behaviour without representation.
New approach, where robots operate in the world,
and use highly reactive architectures, with no
reasoning systems, no manipulable
representations, no symbols, and totally
decentralized computation. (Brooks, 1991)

55
Symbol Grounding and Neural Nets

Alternative idea symbols are grounded in
connectionist representations.
Connectionist symbols distributed
representations patterns of activation across
several units.
Connectionist symbols have internal structure.
They are not meaningless in the same way that
atomic symbols are.
This is a persuasive argument made by Chalmers
(1992)
Chalmers (1992) Subsymbolic computation and the
Chinese Room. In J. Dinsmore (Ed) The symbolic
and connectionist paradigms closing the gap,
Lawrence Erlbaum Hillsdale, New Jersey. pp 25-49.

56
Topics to think about