Artificial Neural Networks ANNs Computing Technology and Applications by Sdhabhon Bhokha Sdhabhonait

1
Artificial Neural Networks (ANNs)Computing
Technology and ApplicationsbySdhabhon Bhokha
(Sdhabhon_at_ait.ac.th http//www.nyakobo.com/)28
June 2000

• ContentIntroduction Artificial Intelligence
  (AI) 2-12Artificial Neural Networks
  (ANNs) 13-28Illustrative Applications 29-42

2
Human Intellectual (How does a man become more
clever?)
3
(No Transcript)
4
(No Transcript)
5
Computer or Hardware

• Von Neumann Computer (Serial Processor)
• It is not definitely compatible with ANNs
  concept
• However, it can be use in ANNs computing (as it
  is still being used today)
• Parallel Processors
• They are fully support the ANNs concept and work
  but only in research work

6
Generalization

• Capability to reasoning or solving even the
  problem which are beyond the learning (or the
  knowledge being).
• Generalization makes the AI different from and
  have edged over all the existing rule-based
  systems.
• Generalization is an ability to respond to
  patterns which are not presented during learning
  or ability to generate unknown situations from
  the known experience or examples, then can derive
  the output for unfamiliar inputs

7
(No Transcript)
8
(No Transcript)
9
Characterization between Neural Network
versusRule-based System (source Klemic, 1993)
Rule-based system
ANNs
Ruleless
Rule-based
Distributed
Symbolic
Parallel
Serial
Discrete
Continuous
Problem-solving
Pattern recognition
Neurophysiology
Psychology
Cognitive
Behavioral
Structural
Functional
Logical
Intuitive
10
Characterization between Neural Network
versusRule-based System (source Klemic, 1993)
ANNs
Rule-based system
Example based
Rule based
Domain free
Domain specific
Finds rules
Needs rules
Little programming needed
Much programming needed
Little programming needed
Difficult to maintain
Not fault tolerant
Easy to maintain
Need (only) a database
Need human expert
Need (only) a database
Rigid Logic
Adaptive system
Requires re-programming
11
(No Transcript)
12
When did ANN start? and How did ANNs start?
13
Five Decades in AI history (Forsyth, 1992)

• 1950s Dark Ages (Neural Networks)
• 1960s The Age of Reason (Automated Logic)
• 1970s The Romantic Movement (Knowledge
  Engineering)
• 1980s The Enlightenment (Machine Learning)
• 1990s Gothic Revival (Neural Nets Revisited!)

14
Conventional Steps of ANNs Modeling

• 1. Data acquisition (only for the supervised
  learning)
• 2. Problem representation (independent/dependent
  variables and forms, i.e. real or binary)
• 3. Architecture and paradigm (layers, nodes,
  links, and learning algorithm)
• 4. Training the network
• 5. Testing for validation
• 6. Application (and routine maintenance)

15
(Expected) Advantages of ANNs Compared to
Conventional Programming (Rule Based System)

• Simple calculation
• Generalization capability
• Fault tolerance
• Fast processing (In application mode)
• Dynamic
• Distribution free

16
Applications of ANNs in Civil Engineering

• Diagnosis,
• Pattern Recognition or Pattern Interpretation,
• Forecasting (Predicting),
• Decision Making,
• Optimization
• Fault Detection

17
Computer Program or ANNs Software

• Commercial software package (various option,
  architecture, paradigms but expensive and not
  flexible for modification)
• Written for specific objectives and functions
  e.g. spreadsheet program, Basic, C, C, PASCAL,
  FORTRAN

18
Awareness of Using ANNs

• Lack of friendliness (No explanation provided),
• Thus, the user needs two background or knowledge
  in 1)problem domain 2) ANNs and 3) computer
  technology .
• Sometimes, developing and maintaining the AI may
  Time-consuming and expensive process, which
  will be the disadvantage.
• There is no guarantee of success on achievement.

19
How Neuro-cell works
Dendrite
Axon
Dendrite
Neuro-cell 1
Neuro-cell 2
20
1
U11
Omk
Tmk
V11
U21
j 1
k 1
2
f(Nbj)
Nbj
U31
i 3
Nbk
f(Nbk)
f(Nb.)
1
0.5
Nb
0
Sigmoid Transformation Function
21
Other Names of ANNS

• ANN s topology or architecture likes humans
  brain, i.e. consists of neuro-cell (or neuron)
• Other names of ANNs are Connectionist Model,
  Parallel Distributed Processing Model,
  Neuromorphic System, Neural Computing

22
Data Acquisition

• Necessary for problem representation and
  modeling, i.e. the inputs, output, topology and
  learning algorithm
• Sources of data are 1) experts questionnaire
  (same as KBES does) 2) historical records and
  3) simulation.
• What should the samples have?
• Adequacy (depended on the parameters to be
  estimated)
• Completeness They can represent characteristics
  and distribution of variables, includes the
  extreme cases of the samples (if possible)

23
Variables

• Independent variables or inputs (real or binary
  and they can be mixed !)
• Dependent variables or outputs (real or binary
  (depended on the transformation function)
• Remark Binary representation use 0 or 1 to
  represents Yes or No, On or Off, Black
  or White, Good or Bad, Easy or Complex

24
ANNs Programming

• Node (or Neuron, Processing Element, Processing
  Unit) acts as a neuro-cell
• Link is the connection between the nodes where
  the input or output can flow through . Two kinds
  of links are 1) fully and 2) patterned link.
• Weight on each connectionist a number (instead of
  hormone or Chemical-Electrical reaction in
  between the axon of a cell and dendrite of the
  other)

25
Hidden Layer (s), Hidden Node(s), and Pruning

• More hidden layer, and nodes tend to increase
  generalization capability of the network but
  learning process may need more samples and
  training time
• Appropriate No. of hidden layer and node can be
  determined by constructive or destructive (or
  pruning) approaches. It is the process by which
  the redundant units can be eliminated from the
  trained larger network to produce the smaller
  network that can perform the same task (at same
  capability) as the larger network
• Not only the hidden layer and node, but also the
  input and output nodes can also be pruned

26
Transformation Function

• Transformation function (which sums up the
  product of inputs and weights, and fire or
  excitate the outputs) Various forms of
  transformation function, e.g. sine, hyperbolic
  tangent, logistic (Sigmoidal, hard limiter) They
  must have at least 2 important characteristics
  1) bounded and 2) differentiable.

27
ANNs Learning Process

• Learning process consists of training and
  testing. The ANNs can learn by means of
  adjusting the weights (At the beginning, the
  weights have to be initiated, i.e. by random
  process). Then, learning process becomes an
  iterative fashion
• Supervised learning (needs adequate samples for
  both training and testing), e.g. Back-propagation
  net and Generalized Delta Rule (BP - GDR)
• Unsupervised learning (does not need the
  samples), sometimes called Self learning),
  e.g. Adaptive

28
Learning Rate and Momentum,

• Learning rate (0-1), in practice use 0.6 to 0.9
  (steps by 0.1)
• Momentum (0-1), steps by 0.1 (to minimize the
  parameters, null momentum is preferred)
• High learning rate and momentum tend to converge
  rapidly but sometimes risk to fall into a local
  minima, and vice versa

29
Criteria to Stop Training

• Nos. of Epoch ( 1 Epoch consists of n cycles
  where n is numbers of training samples)
• Sum Squared Error (SSE) or Mean Squared Error
  (MSE)

30
Function of building
Structural system
Construction Cost Duration
Height
Site accessibility
Complexity of foundation
Exterior finishing
Major Factors Affecting Construction Cost and
Duration of Buildings
31
(No Transcript)
32
Monthly Price Indices for Three Major
Construction Materials during 1987 - 1997
33
Buildings Classified by the Year Start
Constructing (136 projects)
34
Statistics for 136 Building Samples

• 1. Building functions (2-binaries) 63
  residences (46.3), 59 offices (43.4), 6 dual
  functions (4.4) and 8 others (5.9).
• 2. Structural systems (2-binaries) 29
  cast-in-situ RC (21.3), 100 post-tensioned
  slabRC frame(73.5), and 7 others (5.1)
• 3. Height (1-binary) 69 normal height (50.7)
  and 67 extra height (49.3).
• 4. Foundation index (1-binary) 101 complex
  (73.4), 35 simple (25.7). Examples for complex
  foundations are retaining and diaphragm walls,
  and mat footing.

35

• 5. Exterior finishing (2-binaries) 58
  brick/cement block wall (42.7), 77 frameglass
  or cladding (56.6), and 1 prefabricated wall
  (0.7).
• 6. Interior decorating (1-binary) 88 normal
  (64.7), and 48 excellent (35.3). Excellent
  interior decorating consists of premium class of
  1) sanitary fixture, accessories 2) luxury
  electrical accessories 3) carpet, floor tile,
  and wall paper and 4) built-in furniture.
• 7. Site accessibility(1-binary) 43 poor
  (31.6), and 93 good (68.4).
• 8. Price indices (2-real values) average price
  indices for steel and cement are 131 and 126,
  respectively,

36
20,000
Variations in Construction Cost of high-rise
buildings in Greater Bangkok Area during 1987 -
1995 (136 projects)
37
Variation in Construction Duration of High-rise
Buildings in the Greater Bangkok Area during 1987
- 1995 (136 projects)0
38
Description of the Successfully Trained Networks

• Cost Duration
• Input, hidden, output nodes 12, 6, 3 11, 6, 1
• Learning rate (momentum) 0.6 (0) 0.6 (0)
• Training time, sec 1,336 1,885
• Epoch 23,235 44,700
• SSE (training set) 6.28x10-4 7.97x10-6
• SSE (test set) 1.40x10-3 2.11x10-4
• MSE (training set) 0.0030 0.0003
• MSE (test set) 0.0045 0.0018

39
Accuracy of Networks Forecasting (136 samples)

• Cost network
  Duration network
• Underestimated -46.5 to -0.2 0.1 to 136.8
• Nos.of project 58 (42.7) 40 (29.4)
  
• Over estimated 2.4 to 150.0 2.4 to 150
• Nos. of project 78 (57.3) 96 (70.6)
• Average errors
• underestimated -4.8 -4.6
• overestimated 9.2 18.2
• overall 4.4 13.6

40
Variation of Forecasting Errors by 12, 6, 3
Network in according to Construction Costs (136
samples)
41
Variation of Forecasting Errors by 11, 6, 1
Network in according to Construction Duration
(136 samples)
42
Shift of Paradigms (New Fashions)

• Combined ANNs-KBES, i.e. the KBES will be used
  for explanation function
• Combined ANNs-GA, i.e. in analysis, design and
  optimization, the GA will function as learning
  rule
• Combined ANNs-DBMS, i.e. uses DBMS for storing
  the train or test data, routine maintenance