Hybrid Soft Computing for Classification, Control and Prediction Applications

1
Hybrid Soft Computing for Classification,
Control and Prediction Applications

• Piero P. Bonissone
• GE Global Research Center
• Bonissone_at_crd.ge.com

2
Hybrid SC Applications - Outline

• Soft Computing Overview
• Introduction and SC Components
• Hybrid Soft Computing
• Modeling with FL and EA
• Hybrid SC Applications
• Prediction of Paper Web Breakage in Paper Mills
• Digital Underwriting of Insurance Products
• Conclusions

3
Soft Computing

• Soft Computing (SC) the symbiotic use of many
  emerging problem-solving disciplines.

• According to Prof. Zadeh
• "...in contrast to traditional hard computing,
  soft computing exploits the tolerance for
  imprecision, uncertainty, and partial truth to
  achieve tractability, robustness, low
  solution-cost, and better rapport with reality
• Soft Computing Main Components
• Approximate Reasoning
• Probabilistic Reasoning, Fuzzy Logic
• Search Optimization
• Neural Networks, Evolutionary Algorithms

4
Soft Computing Technologies Their Applications
at GE

• Classification
• Monitoring/Anomaly Detection
• Diagnostics
• Prognostics
• Configuration/Initialization
• Prediction
• Quality Assessment
• Equipment Life Estimation
• Scheduling
• Time/Resource Assignments
• Control
• Machine/Process Control
• Process Initialization
• Supervisory Control
• DSS/Auto-Decisioning
• Cost/Risk Analysis
• Revenue Optimization

5
Problem Solving Technologies
6
Hybrid SC Applications - Outline

• Soft Computing Overview
• Introduction and SC Components
• Hybrid Soft Computing
• Modeling with FL and EA
• Hybrid SC Applications
• Prediction of Paper Web Breakage in Paper Mills
• Digital Underwriting of Insurance Products
• Conclusions

7
Soft Computing Probabilistic Systems
Approximate Reasoning
Functional Approximation/ Randomized Search
Evolutionary Algorithms
Multivalued Fuzzy Logics
Neural Networks
Probabilistic Models
Bayesian Belief Nets
Dempster- Shafer
8
Soft Computing Hybrid Probabilistic Systems
Approximate Reasoning
Functional Approximation/ Randomized Search
Evolutionary Algorithms
Multivalued Fuzzy Logics
Neural Networks
Probabilistic Models
Bayesian Belief Nets
Dempster- Shafer
HYBRID PROBABILISTIC SYSTEMS
Probability of Fuzzy Events
Belief of Fuzzy Events
Fuzzy Influence Diagrams
9
Soft Computing Hybrid Probabilistic Systems
Approximate Reasoning
Functional Approximation/ Randomized Search
Evolutionary Algorithms
Multivalued Fuzzy Logics
Neural Networks
Probabilistic Models
Bayesian Belief Nets
Dempster- Shafer
HYBRID PROBABILISTIC SYSTEMS
Probability of Fuzzy Events
Belief of Fuzzy Events
Fuzzy Influence Diagrams
10
Soft Computing Hybrid Probabilistic Systems
Approximate Reasoning
Functional Approximation/ Randomized Search
Evolutionary Algorithms
Multivalued Fuzzy Logics
Neural Networks
Probabilistic Models
Bayesian Belief Nets
Dempster- Shafer
HYBRID PROBABILISTIC SYSTEMS
Probability of Fuzzy Events
Belief of Fuzzy Events
Fuzzy Influence Diagrams
11
Soft Computing FL Systems
Approximate Reasoning
Functional Approximation/ Randomized Search
Evolutionary Algorithms
Multivalued Fuzzy Logics
Neural Networks
Probabilistic Models
Fuzzy Systems
Fuzzy Logic Controllers
12
Soft Computing Hybrid FL Systems
Approximate Reasoning
Functional Approximation/ Randomized Search
Evolutionary Algorithms
Multivalued Fuzzy Logics
Neural Networks
Probabilistic Models
Fuzzy Systems
Fuzzy Logic Controllers
HYBRID FL SYSTEMS
FLC Tuned by NN (Neural Fuzzy Systems)
NN modified by FS (Fuzzy Neural Systems)
FLC Generated and Tuned by EA
13
Soft Computing Hybrid FL Systems
Approximate Reasoning
Functional Approximation/ Randomized Search
Evolutionary Algorithms
Multivalued Fuzzy Logics
Neural Networks
Probabilistic Models
Fuzzy Systems
Fuzzy Logic Controllers
HYBRID FL SYSTEMS
FLC Tuned by NN (Neural Fuzzy Systems)
NN modified by FS (Fuzzy Neural Systems)
FLC Generated and Tuned by EA
14
Soft Computing Hybrid FL Systems
Approximate Reasoning
Functional Approximation/ Randomized Search
Evolutionary Algorithms
Multivalued Fuzzy Logics
Neural Networks
Probabilistic Models
Fuzzy Systems
Fuzzy Logic Controllers
HYBRID FL SYSTEMS
FLC Tuned by NN (Neural Fuzzy Systems)
NN modified by FS (Fuzzy Neural Systems)
FLC Generated and Tuned by EA
15
Soft Computing NN Systems
Functional Approximation/ Randomized Search
Approximate Reasoning
Multivalued Fuzzy Logics
Evolutionary Algorithms
Neural Networks
Probabilistic Models
Recurrent NN
Feedforward NN
Single/Multiple Layer Perceptron
SOM
Hopfield
ART
RBF
16
Soft Computing Hybrid NN Systems
Functional Approximation/ Randomized Search
Approximate Reasoning
Multivalued Fuzzy Logics
Evolutionary Algorithms
Neural Networks
Probabilistic Models
Recurrent NN
Feedforward NN
Single/Multiple Layer Perceptron
SOM
Hopfield
ART
RBF
HYBRID NN SYSTEMS
NN parameters (learning rate h momentum a )
controlled by FLC
17
Soft Computing Hybrid NN Systems
Functional Approximation/ Randomized Search
Approximate Reasoning
Multivalued Fuzzy Logics
Evolutionary Algorithms
Neural Networks
Probabilistic Models
Recurrent NN
Feedforward NN
Single/Multiple Layer Perceptron
SOM
Hopfield
ART
RBF
HYBRID NN SYSTEMS
NN topology /or
weights
generated by EAs
18
Soft Computing EA Systems
Functional Approximation/ Randomized Search
Approximate Reasoning
Multivalued Fuzzy Logics
Evolutionary Algorithms
Neural Networks
Probabilistic Models
Genetic
Algorithms
19
Soft Computing Hybrid EA Systems
Functional Approximation/ Randomized Search
Approximate Reasoning
Multivalued Fuzzy Logics
Evolutionary Algorithms
Neural Networks
Probabilistic Models
Evolution
Genetic
Strategies
Algorithms
Evolutionary
Genetic
Programs
Progr.
HYBRID EA SYSTEMS
EA parameters
EA parameters
(Pop size, select.)
controlled by FLC
controlled by EA
20
Soft Computing Hybrid EA Systems
Functional Approximation/ Randomized Search
Approximate Reasoning
Multivalued Fuzzy Logics
Evolutionary Algorithms
Neural Networks
Probabilistic Models
Evolution
Genetic
Strategies
Algorithms
Evolutionary
Genetic
Programs
Progr.
HYBRID EA SYSTEMS
EA-based search
EA parameters
inter-twined with
(Pop size, select.)
hill-climbing
controlled by EA
21
Soft Computing Hybrid EA Systems
Functional Approximation/ Randomized Search
Approximate Reasoning
Multivalued Fuzzy Logics
Evolutionary Algorithms
Neural Networks
Probabilistic Models
Evolution
Genetic
Strategies
Algorithms
Evolutionary
Genetic
Programs
Progr.
HYBRID EA SYSTEMS
EA-based search
EA parameters
inter-twined with
(Pop size, select.)
hill-climbing
controlled by EA
22
Synergy in SC Reasons Approaches

• Hybrid Soft Computing
• Leverages tolerance for imprecision, uncertainty,
  and incompleteness - intrinsic to the problems to
  be solved
• Generates tractable, low-cost, robust solutions
  to such problems by integrating knowledge and
  data

• Tight Hybridization
• Data-driven Tuning of Knowledge-derived Models
• Translate domain knowledge into initial structure
  and parameters
• Use Global or local data search to tune
  parameters
• Knowledge-driven Search Control
• Use Global or local data search to derive models
  (Structure Parameters)
• Translate domain knowledge into an algorithms
  controller to improve/manage solution convergence
  and quality

23
Synergy in SC Reasons Approaches

• Loose Hybridization (Model Fusion)
• Combine methodologies outputs, not features
• Outputs are compared and aggregated, to increase
  reliability

• Hybrid Search Methods
• Intertwining local search within global search
• Embedding knowledge in operators for global search

24
Hybrid SC Applications

• Soft Computing Overview
• Introduction and SC Components
• Hybrid Soft Computing
• Modeling with FL and EA
• Hybrid SC Applications
• Prediction of Paper Web Breakage in Paper Mills
• Digital Underwriting of Insurance Products
• Conclusions

25
Modeling

• Model
• Structure Parameters Search Method
• Classical control theory
• Structure order of the differential equations
• Parameters coefficients of differential
  equation.
• Search method LMSE, Pole-placement, etc.

26
Soft Computing FL Systems
Approximate Reasoning
Functional Approximation/ Randomized Search
Evolutionary Algorithms
Multivalued Fuzzy Logics
Neural Networks
Probabilistic Models
Fuzzy Systems

• A Mamdani- type FLC approximates a relationship
  between a state X and an output Y by using a KB
  and a reasoning mechanism (generalized
  modus-ponens).
• The Knowledge Base (KB) is defined by
• Scaling factors (SF) ranges of values of state
  and output variables
• Termset (TS) membership functions of values
• Ruleset (RS) a syntactic mapping of symbols from
  X to Y

Fuzzy Logic Controllers
27
Example (MISO) Max-min Composition with
Centroid Defuzzification

• If X is SMALL and Y is SMALL then Z is NEG. LARGE
  
• If X is SMALL and Y is LARGE the Z is NEG. SMALL
• If X is LARGE and Y is SMALL the Z is POS. SMALL
• If X is LARGE and Y is LARGE then Z is POS. LARGE

Response Surface
28
Modeling Using FLC (Mamdani type)

• The structure of the model is the ruleset.
• The parameters of the model are the scaling
  factors and termsets.
• The search method is initialized by knowledge
  engineering and refined with some other external
  methods (SOFC, error minimization, etc.)

29
Soft Computing EA Systems
Functional Approximation/ Randomized Search
Approximate Reasoning
Multivalued Fuzzy Logics
Evolutionary Algorithms
Neural Networks
Probabilistic Models

• Most Evolutionary Algorithms (EAs) can be
  described by
• xt the population at time t
• under representation x
• v is the variation operator(s)
• s is the selection operator

Genetic
Algorithms
xt 1 s(v(xt))
30
Evolutionary Algorithms Scalar-Valued Fitness
Function Optimization

• Example Find the maximum of the function z(x,y)
• z f(x, y) 3(1-x)2exp(-(x2) - (y1)2) -
  10(x/5 - x3 - y5)exp(-x2-y2)
  -1/3exp(-(x1)2 - y2).

31
Modeling Using EA

• Similarly, for EA
• The structure of the model is the representation
  of an individual in the population (e.g., binary
  string, vector, parse tree, Finite State
  Machine).
• The parameters of the model are the Population
  Size, Probability of Mutation, Prob. of
  Recombination, Generation Gap, etc.
• The search method is a global search based on
  maximization of population fitness function

32
Hybrid SC Applications

• Soft Computing Overview
• Introduction and SC Components
• Hybrid Soft Computing
• Modeling with FL and EA
• Hybrid SC Applications
• Prediction of Paper Web Breakage in Paper Mills
• Digital Underwriting of Insurance Products
• Conclusions

33
Soft Computing Applications at GE

• GE Medical Systems
• SPT Auto Analysis for MRI (FL)
• Reverse Engineering of Picker (FL)
• FE Analysis tool (FL)
• X-Ray error Logs Analysis (CBR)

• GE Appliances
• Preferred Service Contracts (Stat.)
• Call Center Support (CBR)

• GE Capital Services
• Mortgage Collateral Evaluation
• (Fusion/FL/CBR)

• GE Aircraft Engines
• Center for Remote Diagn. (CBR)
• Customer Response Center (CBR)
• Anomaly Detection (FL/Stat.)
• IMATE - Maintenance Advisor (NN/FL)
• Resolver Drift - Sensor Fusion (FL)

Engine

• GE Financial Assurance
• GEFA LTC Preferred Customer (Stat./NN)
• GEFA Fixed Life Digital Underwriter
• (Stat, CBR, FL, GA)

• GE Transportation Systems
• Log from Transportation DB (CBR)
• Prototype Train Handling Cntrl. (FL/GA)
• Prototype Trend Analysis (Stat.)
• Embedded/Remote Diagnostics (BBN)

• GE Plastics
• Automated Color Matching (CBR)

• GE Power Gen. Systems
• Remote Anomaly Detection (Stat.)
• Embedded/Remote Diagnostics (BBN)
• Call Center Problem/Solution (CBR)

• LM Fed. Systems
• Scheduling Maintenance for
• Constellation of Satellites (GA)

• GE Industrial Systems
• Paper Web Breakage Prediction
• (NN/Stat./Induction)
• Control Mixing of Cement (FL/GA)

• LM ORSS
• Vessel Management Syst. (AI/GA)

34
Enabling Soft Computing and Related Technologies
35
Example Prediction of Paper Web Breakage
Developed a web breakage propensity indicator and
a time-to-break predictor for the wet-end of a
paper machine (GEIS)
Paper machine
Pulp
Paper
Press section
Dry-end
Wet-end
Water
Water

• Size of a papermaking machines 300 foot long,
  3 floor high
• Size of a paper web 9 to 30 feet
• Traveling speed of a web 4 to 60 MPH

• Typical monthly frequency 35-40 wet-end
  breaks per machine
• Peak values 15 wet-end breaks in a single
  day
• Average production time loss due to web break
  1.6 hours per day
• Average Loss of production due to web
  break 512 of production
• Yearly Revenue Loss Over 6 million per
  machine

36
System Schematics
Data Reduction
Variable Reduction
1
2
3
4
Data
Data
Variable
PCA
segmentation
scrubbing
selection
Value Transformations
Model Generation (1st Indicator)
6
5
8
7
Feature
CART
Filtering
Smoothing
Extraction
Break Indicator 1
Model Generation (2nd Indicator)
10
9
Normali-
Clustering
zation
15
13
14
11
12
Performance
Transform-
ANFIS
Trending
Shuffling
Evaluation
ation
Break Indicator 2
Time to Breaks Prediction
37
Summary of Variable Selection
38
Principal Components Analysis (PCA)
Data size 6,999 by 31
39
The First 3 Principal Components
40
Filtering First 3 Principal Components
41
Feature Extraction of First Three Principal
Components
1st Principal. Comp. 2nd Principal Comp.
3rd Principal Comp.
1st PC

• Features
• 1st Derivative
• 2nd Derivative
• Difference from
• steady state

1st PC 1st Derivative.
1st PC 2nd Deriv
1st PC (Diff from SS)
3rd PC (Diff from SS)
42
Classification Tree Analysis for Web Breakage
Propensity
Last 1/3 (1 hour) break indication Previous
2/3 no break indication Misclassification
10.6FP 14.8FN
43
System Schematics
Data Reduction
Variable Reduction
1
2
3
4
Data
Data
Variable
PCA
segmentation
scrubbing
selection
Value Transformations
Model Generation (1st Indicator)
6
5
8
7
Feature
CART
Filtering
Smoothing
Extraction
Break Indicator 1
Model Generation (2nd Indicator)
10
9
Normali-
Clustering
zation
15
13
14
11
12
Performance
Transform-
ANFIS
Trending
Shuffling
Evaluation
ation
Break Indicator 2
Time to Breaks Prediction
44
Adaptive Network-based Fuzzy Inference Systems
(ANFIS)
Inputs
IF-part
Rules
THEN-part
Output

PCA 1
x1

S
Time to break

x2
PCA 2

45
Trend Analysis Time-to-break against
Predictions. A Stoplight Metaphor
46
Prediction Error at time 60, i.e., E(60)Limits
for Useful Predictions and Results
47
Training and Testing Process
48
Synopsis Of Paper Web Prediction

• 1995-1999
• Created general process to generate Web-breakage
  prediction models
• Based on 1.5 year data from a paper machine in
  Finland
• Scope of project Wet-end breaks with unknown
  causes
• Project transitioned to a Finnish company 1Q
  2000.
• 1999-2000
• Successfully applied to wet-end web-breakage
  prediction of a paper machine in Maine, USA
• 2001
• Successfully applied to wet-end web-breakage
  prediction of a different paper machine in Maine,
  USA

49
Enabling Soft Computing (and Related)
Technologies
50
Example Digital Underwriting for GEFA
Developing an automated system to underwrite 50
of GEFA Fixed Life Applications by December 2,001
Future
GE Proprietary models provide uniformity
confidence in decision
51
Path 1 Clean Cases (Rule Based)
EKG
APS
Summarize APS, EKG
X i
50 variables?
Medical data
Path 2
Underwriter Decision (CBE)
Policy Classification
Path 1
Application Data Received
Order APS? - Decision
Underwriter Decision (RBE)
Preferred Best Preferred Select Standard
Plus Standard Table Rating Decline
Y
X i
App
MIB data
20 variables?
X i
MVR data
Ext data
6 variables?
Paramed Lab data
Establish Confidence Start Digital Underwriting
52
Fuzzy Rule Based Decision Engine (RBDE)
Preferred Best
Preferred Best
Different rule sets
give a Fuzzy Score
1
Preferred
for each rate class
Preferred
Joe Doe
Systolic Blood Pressure
New App
(for under 60 yr old)
1
Info
Systolic Blood Pressure
0.9
(for under 60 yr old)
0
Uniformity of decision

Common
140
155
160
tolerance interpretation
1
Confidence
in decision


degree of
Cholesterol
0
140
145
150
partial satisfaction of constraints

1
used to decide if human Underwriter
Cholesterol
0
needs to supervise decision
0.8
.270
.265
.270

Explanation of decision

1
traceability
to features that violated
0
D
Weight/Height
constraints (audit, compliance, etc.)
.270
275
.280
1

Immediate Applicability

encoding
D
Weight/Height
of Underwriters experience,
0
assessment, and tradeoff policies
6
3
6

DWI
NO
0

Nicotine
NO
6
8
10

Medical Conditions
NO

DWI
NO

Nicotine
NO

Medical Conditions
NO
Aggregation of Partial
Degree of
Aggregation of Partial
Degree of
Satisfaction of Constraints
Placement 0.2
Scores
determine
Aggregation of Partial
Aggregation of Partial
Degree of
best risk
Satisfaction of Constraints
Degree of
Satisfaction of Constraints
classification
Placement 0.8
53
Web-based Fuzzy RBDE
54
Digital Underwriting of Term Life
55
Digital Underwriting of Term Life (cont.)
Met Build Req. for PREFERRED with degree
0.70 (5 lbs over)
56
RBDE Parameter Tuning
Decision Engine Parameter Tuning/Optimization
Evolutionary Optimization Algorithm
Create initial population
Population of trial solutions
Proportional Selection
Intermediate population
(exploitation)
Stochastic Variation
New population
(exploration)
More superior population of trial solutions
57
Decision Engine Parameter Tuning/Optimization
58
Decision Engine Parameter Tuning/Optimization
Computation of Rate Class Decision Mismatch
Penalties
Decision Engine Instance
Individual k
DE Decision
Decision Comparison Correct/Incorrect
Validated Case Decision
Case
All Other Cases
59
RBE Parameter Tuning via Optimization Actuarial
values (NPV) to define penalty weights
Table 1 NPV profit per policy, adjusted for
placement rate (all ages)
Table 2 Lost NPV profit per policy, adjusted
for placement rate (all ages)
60
RBE Parameter Tuning via Optimization RBE
Optimization Results (Before and After)
The Optimization improved the global accuracy
from 90.1 to 93.6
61
GEFA Business Impact

• Instantaneous Decisions
• Faster to Your Customer
• - Enable Creation of New Product
• - Reduce Percentage of Not-Taken policies

• Consistency with Business Rules (Lower UW
  Variability)
• Better Pricing, Higher Margins
• - Possibly Reduced Reinsurance Premium
• - Reduced Risk (Improved Reserve Position)

• Instant Response to Change
• Lead the Market
• - Intellectual Property Protection

• Process Control
• Six Sigma Capability

62
Hybrid SC Applications

• Soft Computing Overview
• Introduction and SC Components
• Hybrid Soft Computing
• Modeling with FL and EA
• Hybrid SC Applications
• Prediction of Paper Web Breakage in Paper Mills
• Digital Underwriting of Insurance Products
• Conclusions

63
Conclusions Soft Computing Applicability

• Applied SC (core related technologies) to a
  broad range of applications, providing
  significant benefits in both cost and growth to
  many GE Businesses
• Classification (Power Gen., Medical systems,
  Aircraft Engine, Transportation)
• Prediction (Industrial Systems)
• Scheduling (Lockheed Martin)
• Control (Industrial Systems, Trading)
• DSS /Auto-Decisioning (GE Financial Assurance,
  Appliances, LM)

• Synergy In Applications of Hybrid SC
• Technology
• Combination of knowledge and data used to derive
  initial structure and to refine parameters via
  local or global search
• Application
• Solution commonality

64
Conclusions SC New Directions

• Present -gt Short Term Future
• SC technologies will widen beyond its current
  constituents.
• Artificial Immune Systems (for Information
  Assurance)
• Fractals (as building blocks in GP or for
  bacteria identification)
• Development of hybrid SC systems with other AI
  paradigms
• EA for model/software update (CBR, RBR, IR)
• Evolutionary software agents
• Push for low-cost solutions / intelligent tools
  will lead to deployment of hybrid SC systems that
  efficiently integrate reasoning and search
  techniques.

• Medium Term Future
• SC technologies are (or will soon be) implemented
  on alternative, non-standard computing mechanisms
• Evolvable Hardware (Field Programmable Gate
  Arrays)
• Bio-inspired Systems DNA and Molecular Computing

65
Conclusions SC Experiments

• Bio-inspired Systems DNA and Molecular Computing
  (Examples)
• Molecular Genetic Programming (Wasiewicz
  Mulawka, 2001)
• Representation of GP graphs by DNA molecules,
  with crossover and negation operators implemented
  using data flow techniques in DNA computing
• DNA-based Fuzzy Systems (Deaton Garzon, 2001)
• Encoding of fuzzy membership functions in Gibbs
  free energy (released upon DNA hybridization),
  leading to the representation of a fuzzy rule set
  (fuzzy associative memory)
• Fuzzy inference performed by modified
  hybridization process
• DNA Neural Network Computation (Mills et al.,
  2001)
• DNA analog neural network in which axons and
  neuron are replaced by diffusion and molecular
  recognition of DNA.
• DNA Evolutionary Computation (Wood et al., 2001)
• Binary-encoded Evolutionary Algorithms, with
  point-wise mutation and crossover, implemented in
  molecular computing, evaluating the OneMax
  fitness function.
• __________________________________________________
  __________________________
• Watson-Crick hybridization of a pair of
  complementary DNA strands makes possible a
  representation of highly parallel selective
  operations that is key for molecular computing
  (Adleman 1994)

66
Questions ?