The Random Matrix Technique of Ghosts and Shadows

- Alan Edelman
- Dept of Mathematics
- Computer Science and AI Laboratories
- Massachusetts Institute of Technology
- (with thanks to Plamen Koev)

Short followed by the Movie

- Some interesting computational techniques
- Random Matrix Theory Theorems, Applications,

and Software - A new application can be more valuable than a

theorem! - A well crafted experiment or package is not a

theorem but it can be as important or even more

to the field! - The Main Show The Method of Ghosts and Shadows

in Random Matrix Theory - Yet another nail in the threefold way coffin

Semi-Circle Law

- Naïve Way
- MATLAB Arandn(n) S(AA)/sqrt(2n)eig

(S) - R
- Amatrix(rnorm(nn),ncoln)S(at(a))/sqrt(

2n)eigen(S,symmetricT,only.valuesT)values - Mathematica ARandomArrayNormalDistribution,n

,nS(ATransposeA)/SqrtnEigenvaluess

Compute All the Eigenvalues

- Sym Tridiagonal ß1real, ß2complex,

ß4quaternion, ß2½?

Diagonals N(0,2), Off-diagonals chi

random-variables N2000 12 seconds vs. 0.2

seconds (factor of 60!!) (Dumitriu E 2002)

Histogram without HistogrammingSturm Sequences

- Count eigs lt 0.5 Count sign changes in
- Det (A-0.5I)1k,1k
- Count eigs in x,xh
- Take difference in number of sign changes at

xh and x

Mentioned in Dumitriu and E 2006, Used

theoretically in Albrecht, Chan, and E 2008

A good computational trick is a good theoretical

trick!

Finite Semi-Circle Laws for Any Beta!

Finite Tracy-Widom Laws for Any Beta!

Stochastic Differential Eigen-Equations

- Tridiagonal Models Suggest SDEs with Brownian

motion as infinite limit - E 2002
- E and Sutton 2005, 2007
- Brian Rider and (Ramirez, Cambronero, Virag,

etc.) - (Lots of beautiful results!)
- (Not todays talk)

Tracy-Widom ComputationsEigenvalues without

the whole matrix!

Never construct the entire tridiagonal

matrix! Just say the upper 10n1/3 by 10n1/3

Compute largest eigenvalue of that, perhaps

using Lanczos with shift and invert strategy! Can

compute for n amazingly large!!!!! And any beta.

E 2003 Persson and E 2005

Other Computational Results

- Infinite Random Matrix Theory
- The Free Probability Calculator (Raj)
- Finite Random Matrix Theory
- MOPS (Ioana Dumitriu) (ß orthogonal polynomials)
- Hypergeometrics of Matrix Argument (Plamen Koev)

(ß distribution functions for finite stats such

as the finite Tracy-Widom laws for Laguerre) - Good stuff, but not today.

Ghosts and Shadows

Scary Ideas in Mathematics

- Zero
- Negative
- Radical
- Irrational
- Imaginary
- Ghosts Something like a commutative algebra of

random variables that generalizes Reals,

Complexes, and Quaternions and inspires

theoretical results and numerical computation

RMT Densities

- Hermite
- c ??i-?jß e-??i2/2 (Gaussian Ensemble)
- Laguerre
- c ??i-?jß ??im e-??i (Wishart Matrices)
- Jacobi
- c ??i-?jß ??im1 ?(1-?i)m2 (Manova Matrices)
- Fourier
- c ??i-?jß (on the complex unit circle) Jack

Polynomials

Traditional Story Count the real parameters

ß1,2,4 for real, complex, quaternion Application

s ß1 All of Multivariate Statistics ß2parked

cars in London, wireless networks

ß4 There and almost nobody cares Dyson 1962

Threefold Way Three Division Rings

ß-Ghosts

- ß1 has one real Gaussian (G)
- ß2 has two real Gaussians (GiG)
- ß4 has four real Gaussians (GiGjGkG)
- ß1 has one real part and (ß-1) Ghost parts

Introductory Theory

- There is an advanced theory emerging (some other

day) - Informally
- A ß-ghost is a spherically symmetric random

variable defined on Rß - A shadow is a derived real or complex quantity

Goals

- Continuum of Haar Measureas generalizing

orthogonal, unitary, symplectic - New Definition of Jack Polynomials generalizing

the zonals - Computations! E.g. Moments of the Haar Measures
- Place finite random matrix theory ßinto same

framework as infinite random matrix theory

specifically ß as a knob to turn down the

randomness, e.g. Airy Kernel - d2/dx2x(2/ß½)dW ?White Noise

Formally

- Let Sn2p/G(n/2)suface area of sphere
- Defined at any n ßgt0.
- A ß-ghost x is formally defined by a function

fx(r) such that ?8 fx(r) rß-1Sß-1dr1. - Note For ß integer, the x can be realized as a

random spherically symmetric variable in ß

dimensions - Example A ß-normal ghost is defined by

f(r)(2p)-ß/2e-r2/2 - Example Zero is defined with constantd(r).
- Can we do algebra? Can we do linear algebra?
- Can we add? Can we multiply?

r0

A few more operations

- ??x?? is a real random variable whose density is

given by fx(r) - (xx)/2 is real random variable given by

multiplying ??x?? by a beta distributed random

variable representing a coordinate on the sphere

Representations

- Ive tried a few on for size. My favorite right

now is - A complex number z with ??z??, the radius and

Re(z), the real part.

Addition of Independent Ghosts

- Addition returns a spherically symmetric object
- Have an integral formula
- Prefer Add the real part, imaginary part

completed to keep spherical symmetry

Multiplication of Independent Ghosts

- Just multiply ??z??s and plug in spherical

symmetry - Multiplication is commutative
- (Important Example Quaternions dont commute,

but spherically symmetric random variables do!)

Shadow Example

- Given a ghost Gaussian, Gß, the length is a real

chi-beta variable ? ß variable.

Linear Algebra Example

- Given a ghost Gaussian, Gß, the length is a real

chi-beta variable ? ß variable. - Gram-Schmidt
- ? ? ?
- or
- Q

Gß Gß Gß

Gß Gß Gß

Gß Gß Gß

?3ß Gß Gß

Gß Gß

Gß Gß

?3ß Gß Gß

?2ß Gß

Gß

?3ß Gß Gß

?2ß Gß

?ß

H3

H2

H1

Gß Gß Gß

Gß Gß Gß

Gß Gß Gß

?3ß Gß Gß

?2ß Gß

?ß

Q has ß -Haar Measure! We have computed

moments! (more later)

Tridiagonalizing Example

Gß Gß Gß Gß

Gß Gß Gß Gß

Gß Gß Gß Gß

Gß Gß Gß Gß Gß

Symmetric Part of

?

Understanding ??i-?jß

- Define volume element (dx) by
- (r dx)rß(dx) (ß-dim volume, like fractals,

but dont really see any fractal theory here) - Jacobians AQ?Q (Sym Eigendecomposition)
- QdAQd?(QdQ)?- ?(QdQ)
- (dA)(QdAQ) diagonal strictly-upper
- diagonal ?d?i (d?)
- off-diag ?((QdQ)ij(?i-?j))(QdQ)

??i-?jß

Haar Measure

- ß1 EQ(trace(AQBQ)k)?C?(A)C?(B)/C?(I)
- Forward Method Suppose you know C?s a-priori.

(Jack Polynomials!) - Let A and B be diagonal indeterminants (Think

Generating Functions) - Then can formally obtain moments of Q
- Example E(q112q222) (na-1)/(n(n-1)(na))
- a2/ ß
- Can Gram-Schmidt the ghosts. Same answers coming

up!

Further Uses of Ghosts

- Multivariate Othogonal Polynomials
- Largest Eigenvalues/Smallest Eigenvalues
- Noncentral Distributions
- Expect Lots of Uses to be discovered