Solution of Sparse Linear Systems

1
Numerical Simulation 219A Spring 2008
Solution of Sparse Linear Systems
Thanks to Kin Sou, Deepak Ramaswamy, Michal
Rewienski, Jacob White, Shihhsien Kuo and Karen
Veroy and especially Luca Daniel
2
Outline of todays lecture

• Solution of Sparse Linear Systems
• Examples of Problems with Sparse Matrices
• Struts and joints, resistor grids, 3-D heat flow
  
• Tridiagonal Matrix Factorization
• General Sparse Factorization
• Fill-in and Reordering
• Graph Based Approach
• Sparse Matrix Data Structures
• Scattering

3
Applications
Sparse Matrices
Space Frame
Nodal Matrix
Space Frame
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
Unknowns Joint positions Equations
forces 0
4
Applications
Sparse Matrices
Resistor Grid
Unknowns Node Voltages Equations
currents 0
5
Applications
Sparse Matrices
Nodal Formulation
Resistor Grid
Matrix non-zero locations for 100 x 10 Resistor
Grid
6
Applications
Sparse Matrices
Nodal Formulation
Temperature in a cube
Temperature known on surface, determine interior
temperature
Circuit Model
7
Outline

• Solution of Dense Linear Systems
• Solution of Sparse Linear Systems
• LU Factorization Reminder.
• Example of Problems with Sparse Matrices
• Struts and joints, resistor grids, 3-d heat flow
  
• Tridiagonal Matrix Factorization
• General Sparse Factorization
• Fill-in and Reordering
• Graph Based Approach
• Sparse Matrix Data Structures
• Scattering

8
Tridiagonal Example
Sparse Matrices
Nodal Formulation
Matrix Form
How many operations to do LU factorization?
m
9
Tridiagonal Example
Sparse Matrices
GE Algorithm
For i 1 to n-1 For each Row For
j i1 to n For each target Row below
the source For k i1 to n
For each Row element beyond Pivot

Order n Operations!
10
Outline

• Solution of Dense Linear Systems
• Solution of Sparse Linear Systems
• LU Factorization Reminder.
• Example of Problems with Sparse Matrices
• Struts and joints, resistor grids, 3-d heat flow
  
• Tridiagonal Matrix Factorization
• General Sparse Factorization
• Fill-in and Reordering
• Graph Based Approach
• Sparse Matrix Data Structures
• Scattering

11
Sparse Matrix Fill-In
Sparse Matrices
Example
Resistor Example
Nodal Matrix
0
Symmetric Diagonally Dominant
12
Fill-In
Sparse Matrices
Example
Matrix Non zero structure
Fill Ins
X
X
X
X
X Non zero
13
Fill-In
Sparse Matrices
Second Example
Fill-ins Propagate
X
X
X
X
X
Fill-ins from Step 1 result in Fill-ins in step 2
14
Fill-In
Sparse Matrices
Reordering
Fill-ins
0
No Fill-ins
Node Reordering - Can reduce fill-in
- Preserves properties (Symmetry, Diagonal
Dominance) - Equivalent to swapping rows
and columns
15
Fill-In
Sparse Matrices
Reordering
Where can fill-in occur ?
Already Factored
Multipliers
16
Fill-In
Sparse Matrices
Reordering
Markowitz Reordering
Greedy Algorithm !
17
Fill-In
Sparse Matrices
Reordering
Why only try diagonals ?

• Corresponds to node reordering in Nodal
  formulation

• Reduces search cost

• Preserves Matrix Properties -
  Diagonal Dominance - Symmetry

18
Fill-In
Sparse Matrices
Pattern of a Filled-in Matrix
Very Sparse
Very Sparse
Dense
19
Fill-In
Sparse Matrices
Unfactored Random Matrix
Symmetric
20
Fill-In
Sparse Matrices
Factored Random Matrix
Symmetric
21
Outline

• Solution of Dense Linear Systems
• Solution of Sparse Linear Systems
• LU Factorization Reminder.
• Example of Problems with Sparse Matrices
• Struts and joints, resistor grids, 3-d heat flow
  
• Tridiagonal Matrix Factorization
• General Sparse Factorization
• Fill-in and Reordering
• Graph Based Approach
• Sparse Matrix Data Structures
• Scattering

22
Matrix Graphs
Sparse Matrices
Construction
Structurally Symmetric Matrices and Graphs
1
X
X
X
X
X
X
X
2
X
X
X
X
3
X
X
X
4
X
X
X
5

• One Node Per Matrix Row
• One Edge Per Off-diagonal Pair

23
Matrix Graphs
Sparse Matrices
Markowitz Products
Markowitz Products
(Node Degree)2
24
Matrix Graphs
Sparse Matrices
Factorization
One Step of LU Factorization
1
X
X
X
X
X
X
X
X
2
X
X
X
X
X
3
X
X
X
4
X
X
X
X
X
5

• Delete the node associated with pivot row
• Tie together the graph edges

25
Matrix Graphs
Sparse Matrices
Example
Graph
Markowitz products ( Node degree)
26
Matrix Graphs
Sparse Matrices
Example
Swap 2 with 1
X
2
1
1
2
X
Eliminate 1
27
Graphs
Sparse Matrices
Resistor Grid Example
Unknowns Node Voltages Equations
currents 0
28
Matrix Graphs
Sparse Matrices
Grid Example
29
What should you pivot for?
Sparse Matrices
For growth control? Or to avoid fill-ins?
A) LU factorization applied to a strictly
diagonally dominant matrix will never produce a
zero pivot
B) The matrix entries produced by LU
factorization applied to a strictly diagonally
dominant matrix will never increase by more than
a factor 2(n-1) which is anyway the best you can
do by pivoting for growth control
Bottom line if your matrix is strictly
diagonally dominant no need for numerical pivot
for growth control!
30
Sparse Factorization Approach

• Assume matrix requires NO numerical pivoting.
• Diagonally dominant or symmetric positive
  definite.
• Pre-Process
• Use Graphs to Determine Matrix Ordering
• Many graph manipulation tricks used.
• Form Data Structure for Storing Filled-in Matrix
• Lots of additional non-zero added
• Put numerical values in Data Structure and factor
• Computation must be organized carefully!

31
Summary of Sparse Systems

• LU Factorization and Diagonal Dominance.
• Factor without numerical pivoting
• Sparse Matrices
• Struts, resistor grids, 3-d heat flow -gt O(N)
  non-zeros
• Tridiagonal Matrix Factorization
• Factor in O(N) operations
• General Sparse Factorization
• Markowitz Reordering to minimize fill
• Graph Based Approach
• Factorization and Fill-in
• Useful for estimating Sparse GE complexity

32
Outline

• Solution of Sparse Linear Systems
• Example of Problems with Sparse Matrices
• Tridiagonal Matrix Factorization
• General Sparse Factorization
• Fill-in and Reordering
• Graph Based Approach
• Summary of the Algorithm
• Sparse Matrix Data Structures
• Scattering and symbolic fill in

33
Sparse Data Structure
Sparse Matrices
Vector of row pointers
Arrays of Data in a Row
Matrix entries
Val 11
Val 12
Val 1K
1
Column index
Col 11
Col 12
Col 1K
Val 21
Val 22
Val 2L
Col 21
Col 22
Col 2L
Row pointers Column Indices
Goal Never store a 0 Never multiply by 0
Val N1
Val N2
Val Nj
N
Col N1
Col N2
Col Nj
34
Sparse Data Structure
Sparse Matrices
Problem of Misses
Eliminating Source Row i from Target row j
Row i
Row j
Must read all the row j entries to find the 3
that match row i
35
Sparse Data Structure
Sparse Matrices
Data on Misses
More misses than ops!
Every Miss is an unneeded Memory Reference!
36
Sparse Data Structure
Sparse Matrices
Scattering for Miss Avoidance
Row j
Use target row approach Row j is the target. 1)
Read all the elements in Row j, and scatter them
in an n-length vector 2) Access only the needed
elements using array indexing!
37
Sparse Matrices Another Data Structure
Orthogonal linked list
But if fill in occurs, pointer structures change.
What do we do?
Pre-compute pivoting order and sparse fill in
structure symbolically
38
Summary of Sparse Systems

• LU Factorization and Diagonal Dominance.
• Factor without numerical pivoting
• hard problems (ill-conditioned)
• Sparse Matrices
• Struts, resistor grids, 3-d heat flow -gt O(N)
  nonzeros
• Tridiagonal Matrix Factorization
• Factor in O(N) operations
• General Sparse Factorization
• Markowitz Reordering to minimize fill
• Graph Based Approach
• Factorization and Fill-in
• Useful for estimating Sparse GE complexity
• Sparse Data Structures
• Scattering and symbolic fill in