Back to Basics: Classification and Inference Based on Input Feedback Structure

1
Back to Basics Classification and Inference
Based on Input Feedback Structure

• Tsvi Achler
• Eyal Amir

Department of Computer Science University of
Illinois at Urbana-Champaign
2
AI -gt AGI

• Ability to generalize
• Even if only learned basics
• Training distribution ? test distribution
• Avoid Combinatorial Explosion
• Allows complex networks

3
New Basic Computational Structure

• Based on massive feedback to inputs
• No emphasis on weight parameters
• Input Feedback during testing

4
Avoids Combinatorial Explosionvia Simple
Connectivity
Y1
Y3
Y4
Output Nodes
I4
Input Nodes
x4
x1
x2
x3
Connections
Positive
Negative
5
Iterative
Y2
Y1
I2
I1
x1
x2
1
thus Wy
6
Back
Y2
Y1
I2
I1
x1
x2
7
Forward
Y2
Y1
I2
I1
x1
x2
8
Back
Y2
Y1
I2
I1
x1
x2
9
Active (1)
Y2
Y1
Inactive (0)
I2
I1
10
Active (1)
C2
Inactive (0)
I2
I1
11
Active (1)
C2
Inactive (0)
I2

12
Active (1)
Inactive (0)


13
Active (1)
Inactive (0)


14
Active (1)
Inactive (0)
I2

15
Active (1)
Inactive (0)


16
Steady State
Graph of Dynamics
1

Y1
Y2
Activity
0
I2
0
1
3
2
4
5
I1
Simulation Time (T)
17
Resolving Pattern Interactions
Network Configuration
Steady State
Results Node?Value
1

2

Inputs
Node


Input
A

B
(0, ½)
A1 and B ½ (
1,
Half Activation
Half Response
½)
A.12 C1.12 A2.5 B1 c11.5 c21
18
Resolving Pattern Interactions
Based on Available Representations
2

3

Cells

Inputs
A

B

C

19
Resolving Pattern Interactions
Binding
Most efficient configuration
Not possible with OvA or AvA
20
Can be Chained Ad Infinitum
2

3


N
Nodes
...
Inputs
N

O
A

B

C


N
1

2

3

Nodes

...
Inputs
N

O
A

B

C

21
New data Recognize Scene When Trained on
Individuals

• Teach single letters
• Test multiple simultaneous letters
• A scene is beyond the training distribution

22
Feature Extraction

• Bag-of-features

512 features

• Found in Visual Cortex
• Pixels Separated into features

Inputs to Model
In
I3
Feature Examples
Feature 1 Feature 2 Feature 3 Feature 4 Feature
n
Feature 1
Feature 2
Feature 3

?
D X
Fig 4 Simple feature extractor presenting
non-spatial information from visual field
collective pixel patterns presented to
network
23
Two Stimuli Simultaneously

• A B

100
90
80
70
60
of combinations
50
40
30
20
10
0
0/2
1/2
2/2
Letters Correctly Classified
Figure 5 NN with two letter retraining
24
Four Stimuli Simultaneously
A B C D

• i.e.
• (A B C D)

100
90
80
70
60
50
of combinations
40
30
20
10
0
0/4
1/4
2/4
3/4
4/4
Letters Correctly Classified
Figure 6 Four Letter Classification
Figure 5 NN with two letter retraining
25
Difficulty

• Nonlinear Equations
• Cant mathematically prove general properties

26
Steps Towards AGI

• Generalize Outside Training Distribution
• Structure Avoids Combinatorial Explosion

27
Acknowledgements

• Cyrus Omar

National Geospatial-Intelligence Agency
HM1582-06--BAA-0001
28
Equations
Activation
Feedback
Inhibition
Combined
C collection of all output cells Ca cell
a. Na the set of input connections to cell
Ca. na the number of processes in set Na of cell
Ca. P primary inputs (not affected by shunting
inhibition). I collection of all inputs Ib input
cell b. Mb the set of recurrent feedback
connections to input Ib. mb the number of
connections in set Mb
Q shunting inhibition. Qb shunting inhibition at
input b.