Smoothed Analysis of Algorithms

1
Smoothed Analysis of Algorithms

• Shang-Hua Teng
• Boston University
• Akamai Technologies Inc

Joint work with Daniel Spielman (MIT)
2
Outline

• Part I Introduction to Algorithms
• Part II Smoothed Analysis of Algorithms
• Part III Geometric Perturbation

3
Part I Introduction to Algorithms

• Type of Problems
• Complexity of Algorithms
• Randomized Algorithms
• Approximation
• Worst-case analysis
• Average-case analysis

4
Algorithmic Problems

• Decision Problem
• Can we 3-color a given graph G?
• Search Problem
• Given a matrix A and a vector b, find an x
• s.t., A x b
• Optimization Problem
• Given a matrix A and a vector b, and an
  objective vector c, find an x that
• maximize cT x
• s.t. A x b

5
The size and family of a problem

• Instance of a problem
• Input
• for example, a graph, a matrix, a set of points
• desired output
• yes,no, coloring, solution-vector, convex hull
• Input and output size
• Amount of memory needed to store the input and
  output
• For example number of vertices in a graph,
  dimensions of a matrix, the cardinality of a
  point set
• A problem is a family of instances.

6
An Example

• Median of a set of numbers
• Input a set of numbers a1,, a2,, an
• Output ai that maximizes

1 2 3 4 5 6 7 8 9 10
7
Quick Selection

• Quick Selection (a1,, a2,, an, k )
• Choose a1 in a1,, a2,, an
• Divide the set into
• Cases
• If k L 1, return a1
• If k lt L, recursively apply Quick_Selection(L,k)
  
• If k gt L1, recursively apply
  Quick_Selection(L,n-k-1)

8
Worse-Case Time Complexity

• Let T(a1,, a2,, an ) be the number of basic
  steps needed in Quick-Selection for input a1,,
  a2,, an.
• We classify inputs by their size
• Let An be the set of all input of size n.
• T(n) n2 - n

9
Better Algorithms from Worst-case View point

• Divide and Conquer
• Linear-time algorithm
• Blum-Floyd-Pratt-Rivest-Tarjan

10
Average-Case Time Complexity

• Let An be the set of all input of size n.
• Choose a1,, a2,, an uniformly at random
• E(T(n)) O(n)

11
Randomized Algorithms

• Quick Selection (a1,, a2,, an, k )
• Choose a random element s in a1,, a2,, an
• Divide the set into
• Cases
• If k L 1, return s
• If k lt L, recursively apply Quick_Selection(L,k)
  
• If k gt L1, recursively apply
  Quick_Selection(L,n-k-1)

12
Expected Worse-Case Complexityof Randomized
Algorithms
E(T(n)) O(n)
13
Approximation Algorithms

• Sampling-Selection (a1,, a2, , an)
• Choose a random element ai from a1,, a2,, an
• Return ai.
• ai is a dmedian if
• Probai is a (1/4)median 0.5

14
Approximation Algorithms

• Sampling-Selection (a1,, a2, , an)
• Choose a set S of random k elements from
  a1,,, an
• Return the median ai of S.
• Complexity O(k)

15
When k 3
16
Iterative Middle-of-3(Miller-Teng)
Randomly assign elements from a1,, a2, , an
to the leaves
17
Summary

• Algorithms and their complexity
• Worst-case complexity
• Average-cast complexity
• Design better worse-case algorithm
• Design better algorithm with randomization
• Design faster algorithm with approximation

18
Sad and Exciting Reality

• Most interesting optimization problems are hard
• P vs NP
• NP-complete problems
• Coloring, maximum independent set, graph
  partitioning
• Scheduling, optimal VLSI layout, optimal
  web-traffic assignment, data mining and
  clustering, optimal DNS and TCPIP protocols,
  integer programming
• Some are unknown
• Graph isomorphism
• factorization

19
Good News

• Some fundamental problems are solvable in
  polynomial time
• Sorting, selection, lower dimensional
  computational geometry
• Matrix problems
• Eigenvalue problem
• Linear systems
• Linear programming (interior point method)
• Mathematical programming

20
Better News I

• Randomization helps
• Testing of primes (essential to RSA)
• VC-dimension and sampling for computational
  geometry and machine learning
• Random walks various statistical problems
• Quicksort
• Random routing on parallel network
• Hashing

21
Better News II

• Approximation algorithms
• On-line scheduling
• Lattice basis reduction (e.g. in cryptanalysis)
• Approximate Euclidean TSP and Steiner trees
• Graph partitioning
• Data clustering
• Divide-and-conquer method for VLSI layout

22
Real Stories

• Practical algorithms and heuristics
• Great-performance empirically
• Used daily by millions and millions of people
• Worked routinely from chip design to airline
  scheduling
• Applications
• Internet routing and searching
• Scientific simulation
• Optimization

23
PART II Smoothed Analysis of Algorithms

• Introduction of Smoothed Analysis
• Why smoothed analysis?
• Smoothed analysis of the Simplex Method for
  Linear Programming

24
Smoothed Analysis of Algorithms Why The Simplex
Method Usually Takes Polynomial Time
Gaussian Perturbation with variance s2

• Daniel A. Spielman (MIT)
• Shang-Hua Teng (Boston University)

25
Remarkable Algorithms and Heuristics

• Work well in practice, but
• Worst case bad,
• exponential,
• contrived.
• Average case good,
• polynomial,
• meaningful?

26
Random is not typical
27
Smoothed Analysis of Algorithms
worst case maxx T(x) average case avgr
T(r) smoothed complexity maxx avgr T(xer)
28
Smoothed Analysis of Algorithms

• Interpolate between Worst case and Average Case.
• Consider neighborhood of every input instance
• If low, have to be unlucky to find bad input
  instance

29
Smoothed Complexity
30
Classical Example Simplex Method for Linear
Programming
max zT x s.t. A x y

• Worst-Case exponential
• Average-Case polynomial
• Widely used in practice

31
The Diet Problem
Carbs Protein Fat Iron Cost
1 slice bread 30 5 1.5 10 30
1 cup yogurt 10 9 2.5 0 80
2tsp Peanut Butter 6 8 18 6 20
US RDA Minimum 300 50 70 100
Minimize 30 x1 80 x2 20 x3 s.t. 30x1
10 x2 6 x3 ? 300 5x1
9x2 8x3 ? 50 1.5x1 2.5 x2
18 x3 ? 70 10x1
6 x3 ? 100
x1, x2, x3 ? 0
32
Linear Programming
max zT x s.t. A x y
Max x1 x2 s.t x1 1
x2 1 -x1 - 2x2 1

33
Smoothed Analysis of Simplex Method
G is Gaussian
34
Smoothed Analysis of Simplex Method

• Worst-Case exponential
• Average-Case polynomial
• Smoothed Complexity polynomial

max zT x s.t. aiT x 1, ai 1
max zT x s.t. (aisgi )T x 1
35
Perturbation yields Approximation

• For polytope of good aspect ratio

36
But, combinatorially
37
The Simplex Method
38
History of Linear Programming

• Simplex Method (Dantzig, 47)
• Exponential Worst-Case (Klee-Minty 72)
• Avg-Case Analysis (Borgwardt 77, Smale 82,
  Haimovich, Adler, Megiddo, Shamir, Karp, Todd)
• Ellipsoid Method (Khaciyan, 79)
• Interior-Point Method (Karmarkar, 84)
• Randomized Simplex Method (mO(?d) )
• (Kalai 92, Matousek-Sharir-Welzl
  92)

39
Shadow Vertices
40
Another shadow
41
Shadow vertex pivot rule
start
z
objective
42
Theorem For every plane, the expected size of
the shadow of the perturbed tope is poly(m,d,1/s )
43
Theorem For every z, two-Phase Algorithm runs in
expected time poly(m,d,1/s )
z
44
A Local condition for optimality
z
a2
z
0
Vertex on a1,,ad maximizes z iff z Î
cone(a1,,ad )
a1
45
Primal a1T x ? 1 a2T x ? 1 amT x ?
1
Polar ConvexHull(a1, a2, am)
46
(No Transcript)
47
Polar Linear Program
z
max ? ?z Î ConvexHull(a1, a2, ..., am)
48
Initial Simplex
Opt Simplex
49
Shadow vertex pivot rule
50
(No Transcript)
51
Count facets by discretizingto N directions, N ??
52
Count pairs in different facets
So, expect c Facets
53
Expect cone of large angle
54
Angle
Distance
55
Isolate on one Simplex
56
Integral Formulation
57
Example For a and b Gaussian
distributed points, given that ab
intersects x-axis Prob? lt
e O(e2)
a
?
b
58
a
?
b
59
(No Transcript)
60
(No Transcript)
61
a
62
(No Transcript)
63
Change of variables
u
a
z
?
v
b
da db (uv)sin(q) du dv dz dq
64
Analysis For e lt e0, Pe lt e2
Slight change in q has little effect on ni
for all but very rare u,v,z
65
Distance Gaussian distributed corners
66
Idea fix by perturbation
67
Trickier in 3d
68
Future Research Simplex Method

• Smoothed analysis of other pivot rules
• Analysis under relative perturbations.
• Trace solutions as un-perturb.
• Strongly polynomial algorithm for linear
  programming?

69
A Theory Closer to Practice

• Optimization algorithms and heuristics, such as
  Newtons Method, Conjugate Gradient, Simulated
  Annealing, Differential Evolution, etc.
• Computational Geometry, Scientific Computing and
  Numerical Analysis
• Heuristics solving instances of NP-Hard
  problems.
• Discrete problems?
• Shrink intuition gap between theory and practice.

70
Part III Geometric Perturbation

• Three Dimensional Mesh Generation

71
Delaunay Triangulations for Well-Shaped 3D Mesh
Generation

• Shang-Hua Teng
• Boston University
• Akamai Technologies Inc.

72
Collaborators Siu-Wing Cheng, Tamal Dey,
Herbert Edelsbrunner, Micheal Facello Damrong
Guoy Gary Miller, Dafna Talmor, Noel
Walkington Xiang-Yang Li and Alper Üngör
73
3D Unstructured Meshes
74
Surface and 2D Unstructured Meshes
courtesy NASA
courtesy N. Amenta, UT Austin
courtesy Ghattas, CMU
75
Numerical Methods
Formulation MathEngineering
Domain, Boundary, and PDEs
ad hoc
Point Set Triangulation
Mesh Generation geometric structures
octree
Delaunay
element
difference
Approximation Numerical Analysis
Finite
volume
Linear System algorithm data structures
Axb
multigrid
direct method
iterative method
76
Outline

• Mesh Generation in 2D
• Mesh Qualities
• Meshing Methods
• Meshes and Circle Packings
• Mesh Generation in 3D
• Slivers
• Numerical Solution Control Volume Method
• Geometric Solution Sliver Removal by Weighted
  Delaunay Triangulations
• Smoothed Solution Sliver Removal by Perturbation

77
Badly Shaped Triangles
78
Aspect Ratio (R/r)
79
Meshing Methods
The goal of a meshing algorithm is to generate a
well-shaped mesh that is as small as possible.

• Advancing Front
• Quadtree and Octree Refinement
• Delaunay Based
• Delaunay Refinement
• Sphere Packing
• Weighted Delaunay Triangulation
• Smoothing by Perturbation

80
Balanced Quadtree Refinements(Bern-Eppstein-Gilb
ert)
81
Quadtree Mesh
82
Delaunay Triangulations
83
Why Delaunay?

• Maximizes the smallest angle in 2D.
• Has efficient algorithms and data structures.
• Delaunay refinement
• In 2D, it generates optimal size, natural looking
  meshes with 20.7o (Jim Ruppert)

84
Delaunay Refinement(Jim Ruppert)
2D insertion
1D insertion
85
Delaunay Mesh
86
Local Feature Spacing (f)
The radius of the smallest sphere centered at a
point that intersects or contains two
non-incident input features
87
Well-Shaped Meshes and f
88
f is 1-Lipschitz and Optimal
89
Sphere-Packing
90
b -Packing a Function f
f(p)/2
p
q

• No large empty gap the radius of the largest
  empty sphere passing q is at most b f(q).

91
The Packing Lemma (2D)(Miller-Talmor-Teng-Walkin
gton)

• The Delaunay triangulation of a b -packing is a
  well-shaped mesh of optimal size.
• Every well-shaped mesh defines a b -packing.

92
Part I Meshes to Packings
93
Part II Packings to Meshes
94
3D Challenges

• Delaunay failed on aspect ratio
• Quadtree becomes octree (Mitchell-Vavasis)
• Meshes become much larger
• Research is more Challenging!!!

95
Badly Shaped Tetrahedra
96
Slivers
97
Radius-Edge Ratio(Miller-Talmor-Teng-Walkington)
R/L
98
The Packing Lemma (3D)(Miller-Talmor-Teng-Walkin
gton)

• The Delaunay Triangulation of a b -packing is a
  well-shaped mesh (using radius-edge ratio) of
  optimal size.
• Every well-shaped (aspect-ratio or radius-edge
  ratio) mesh defines a b -packing.

99
Uniform Ball Packing

• In any dimension, if P is a maximal packing of
  unit balls, then the Delaunay triangulation of P
  has radius-edge at most 1.

e is at least 2
r
r is at most 2
100
Constant Degree Lemma (3D)(Miller-Talmor-Teng-Wa
lkington)

• The vertex degree of the Delaunay triangulation
  with a constant radius-edge ratio is bounded by a
  constant.

101
Delaunay Refinement in 3D Shewchuck
102
Slivers
103
Sliver the geo-roach
104
Coping with Slivers Control-Volume-Method(Miller
-Talmor-Teng-Walkington)
105
Sliver Removal by Weighted Delaunay
(Cheng-Dey-Edelsbrunner-Facello-Teng)
106
Weighted Points and Distance
z
p
107
Orthogonal Circles and Spheres
108
Weighted Bisectors
109
Weighted Delaunay
110
Weighted Delaunay and Convex Hull
111
Parametrizing Slivers
D
L
Y
112
Interval Lemma

• Constant Degree The union of all weighted
  Delaunay triangulations with Property r and
  Property 1/3 has a constant vertex degree

113
Pumping Lemma(Cheng-Dey-Edelsbrunner-Facello-Ten
g)
z
P
p
H
D
q
Y
r
s
114
Sliver Removal by Flipping

• One by one in an arbitrary ordering
• fix the weight of each point
• Implementation flip and keep the best
  configuration.

115
Experiments (Damrong Guoy, UIUC)
116
Initial tetrahedral mesh 12,838 vertices, all
vertices are on the boundary surface
117
Dihedral angle lt 5 degree 13,471
118
Slivers after Delaunay refinement 881
119
Slivers after sliver-exudation 12
120
1183 slivers
15,503 slivers
142 slivers, less elements, better distribution
121
563 slivers
5636 slivers
Triceratops
18 slivers, less elements, better distribution
122
Heart
4532 slivers
173 slivers
1 sliver, less elements, better distribution
123
Smoothing and Perturbation

• Perturb mesh vertices
• Re-compute the Delaunay triangulation

124
Well-shaped Delaunay Refinement(Li and Teng)

• Add a point near the circumcenter of bad element
• Avoids creating small slivers
• Well-shaped meshes