The Incorporation of Prior Knowledge and Constraints through Bayesian Analysis into a NeuroFuzzy Inf

1
The Incorporation of Prior Knowledge and
Constraints through Bayesian Analysis into a
Neuro-Fuzzy Inference System Framework

• Tom Musicka
• Centre for Process Analytics and Control
  Technology
• University Of Newcastle
• Contact tom.musicka_at_ncl.ac.uk

2
Introduction

• Problem Definitions
• Data Analysis
• PreScreening
• Data Validation
• Neuro-Fuzzy Modelling
• Model Comparison
• Results
• Bayesian Framework
• Results
• Conclusions

3
Data Mining

• Data Capture and Treatment
• Analysis of observational data sets to find
  unsuspecting relationships and to summarise the
  data in a number of understandable and useful
  ways
• Prediction
• Classification
• Detection of Relationships
• Explicit Modelling
• Clustering
• Deviation Detection

4
Corus - Process and Scheduling Description

• Hot Strip Steel Rolling - Corus
• Multigrade Operation
• Multistage Process
• Reheating
• Multi-Pass Reversing Rougher
• Multi-Pass Finishing Rolling
• Run Out Table (ROT) Cooling
• Coiling
• Testing

5
Corus - Problem Definition

• Virtual Test House
• Mechanical Property Prediction
• Ultimate Tensile Strength
• Yield Stress
• Elongation
• Project Description
• To produce a modelling system capable of
  accurately and consistently predicting the the
  mechanical properties of Steel rolled at Hot
  Strip Mills within Corus UK.

6
Shell - Process and Scheduling Description

• Refinery High Viscosity Index (HVI) Operation -
  Shell
• Multicrude Operation
• Continuous and Batch Operations
• Pressure to Minimise Working Capital
• Statistical Visualisation
• Masters Thesis
• Four Unit Process
• High Vacuum Unit (HVU)
• Furfural Extraction Unit (FEU)
• Propane De-Asphalting Unit (PDU)
• MEK De-Waxing Unit (MDU)

7
Shell - Problem Definition

• Slack Wax Oil Content (SWOC) Prediction
• MDU Produces Lube Oil and Slack Wax
• Slack Wax Contains Oil
• Useful to know the Oil Content
• Project Description
• To produce a model capable of accurately and
  consistently predicting the SWOC. This Model will
  be considered within the feasibility report for
  the purchase of an online physical measurement
  system

8
Data Characteristics

• Multiple Input, Single Output Models
• Discrete and Continuous Variables
• Missing Data
• Process Noise
• Outliers
• Multiple Operating Regions
• Strong Clustering
• Overlapping Regions
• Local Modelling Techniques
• Non-Linear Relationships

9
Data Screening

• Univariate Checks
• Time Series Visualisation
• Multivariate Statistics
• Principal Component Analysis
• Outlier Detection
• Process Visualisation
• Multi-Modal Data
• Local PCA Models
• Process Grades
• Fuzzy Clustering
• MixPCA

10
Data Analysis - Principal Component Analysis
11
Local Principal Component Analysis
12
Data Splitting

• Data is Divided into 3 Data Sets
• Training
• Data to Build Model
• Validation
• Data used to Define Training Stopping Epoch
• Improve Generalisation
• Testing
• Data Used to Quote Model Results
• Unseen by Model in Training Phase
• Ratio 701515

13
Data Splitting

• Outliers Removed
• Fully Populated Data Set
• Require Representation of Full Scope of Process
  Within Training Data
• Random Selection
• Sample from Data Distribution
• With Respect to Operating Region
• Kennard-Stone Selection
• Sample from Flat Distribution
• Mahalanobis Distance Metric for Selection
• Analysis for Each Operating Region
• Ensures Training Set Covers Full Scope

14
Advanced Modelling Studies

• Projection to Latent Structures
• Linear
• Predictive Extension of PCA
• Neural Networks
• Non-linear
• Black-Box
• Neuro-Fuzzy Inference Systems
• Adaptive Neuro-Fuzzy Inference System - ANFIS
• Fuzzy Rulebase - Model Switching / Merging
• Local Linear Models
• Compromise between Accuracy, Interpretation and
  Simplicity

15
ANFIS Structure
16
Corus - Model Comparison Results
17
Shell - Model Comparison Results
18
Model Comparison Results - Corus and Shell
19
Bayesian Methods

• Mathematical Foundation to Model Optimisation
• Incorporates Use of Prior Knowledge
• Advantage over Maximum Likelihood Models
• Bayes Rule
• Posterior (Likelihood Prior)/Evidence
• Evidence is Constant for Given Model Structure

20
Bayesian Methods

• Posterior - p(wD,H)
• Probability of Parameters, w, Representing Data,
  D, for Given Model, H
• Likelihood - p(Dw,H)
• Probability of the Model Fitting the Data
• Accounts for Least Squares Modeling Power
• Prior - p(wH)
• Incorporates Prior Knowledge into Training
• Model Constraints
• Independent of Data

21
Bayesian Methods

• Bayesian Model Averaging
• Accounts for Model Uncertainty
• Avoids Use of Single Model
• Prior Distribution
• Incorporation of Process Knowledge
• Favour Certain Characteristics
• Model Interpretability
• Model Accuracy

22
Bayesian ANFIS - BANFIS

• Single ANFIS Structure
• Process Knowledge
• Cluster Analysis
• Bayesian Model Averaging
• Sample Rulebase Parameters
• Markov Chain Monte Carlo
• Rejection Sampling
• Single Local Model Learning Stage
• Cover Important Parameter Space

23
BANFIS - Results Comparison - Corus
24
BANFIS - Results Comparison - Corus
25
BANFIS - Results Comparison - Shell
26
BANFIS - Results Comparison - Shell
27
BANFIS - Advantages and Disadvantages

• Advantages
• Improved results
• Incorporates Parameter Uncertainty
• Avoids Optimisation Training Problems
• Avoids Single Network Solution
• Incorporation of Prior Knowledge
• Disadvantages
• Complex Hierarchical Analysis
• Network Interpretation Issues

28
Conclusions

• Data Validation Methods
• Screening
• Local Models
• Essential for Non-linear Modelling Methods
• Compromise between
• Accuracy
• Interpretation
• Model Complexity
• Results
• Improvements in Accuracy and Data Analysis
• Implementation of Bayesian Framework

29
Acknowledgements

• CPACT - School of Chemical Engineering
• TCD
• Paul Kitson - Corus UK
• Phil Jonathan, Paul Blackhurst - Statistics and
  Risk, Shell Global Solutions