Ocean Circulation: Sink or Swim

1
COMPUTATIONAL SCIENCES BROWN BAG SEMINAR
Ocean Circulation Sink or Swim
Danny Thorne
UK Department of Computer Science
Tuesday, April 17, 2001
Supported in part by Sandia National Laboratory,
Lawrence Livermore National Laboratory, and the
National Science Foundation. Nuke em if they
cant take a joke.
2
Craig C. Douglas University of Kentucky and Yale
University
Gundolf Haase Johannes KeplerUniversity-Linz

• Mohamed Iskandarani
• University of Miami

Danny Thorne University of Kentucky and Sandia
National Laboratory
Michael Sheetz University of Kentucky
3
Outline
This talk chronicles the

• Past
• Background
• Single Layer
• Multiple Layer
• Present
• Subgrid Structuring Techniques
• Simultaneous Iteration
• Lumped Schur Complement
• Future
• C
• 3D
• Non-Hydrostatic

of the Spectral Element Ocean Model (SEOM).
4
(No Transcript)
5
(No Transcript)
6
Difficulties of Ocean Modeling

• Stratification
• Rotation
• Complicated basins
• Small aspect ratio (depth ltlt L)
• Turbulence (jets, eddies, fronts)
• Multiple time and length scales
• Poorly known I.C. and B.C.

7
Advantages of Spectral Elements

• h-p finite element method
• Dual convergence paths
• Geometric Flexibility
• Unstructured grids
• Accuracy
• Low numerical dispersion and dissipation errors
• Scalability
• Dense Computational Kernels, O(KN3)
• Sparse neighbor-neighbor communications

8
(No Transcript)
9
Unstructured Grids

• Complicated ocean basins
• No masking
• Local (elemental) mapping only
• Multiscale Simulations
• Increase resolution where needed
• Regional-Global simulations possible in single
  setting
• Avoid open boundary conditions

10
North East Pacific Grid
11
Grid Spacing (km)
12
Elements Collocation points
13
Disadvantages(TANSTAAFL!)

• Stricter CFL stability than FD
• Gibbs oscillations near fronts
• Global conservation only
• More CPU intensive than FD/grid point

14
SEOM Versions and Applications

• Single Layer
• No stratification effects
• Solution is the depth-mean flow
• 1.5 layer (wind circulation/abyssal flow)
• Global long period tides
• Estuarine modeling
• Multiple Layers
• Vertical coordinate is density (isopycnal)
• Vertical stack of single layers
• Wind driven circulation, 2-5 layers (soon up
  to16)

15
Motivation for Layered SEOM

• Mathematically simpler than SEOM-3D
• Computationally simpler and faster
• No cross isopycnal diffusion
• No pressure gradient errors
• Baroclinic processes possible with 2 layers
• Eddy resolving simulations can be produced
• relatively easily and cheaply

16
Current Work
/ TODO Outline /
17
Filtering

• Each layer has to solve
• , filtered vorticity
• , filtered divergence field
• The filtering is done by series expansion and the
  Boyd-Vandeven filter in each spectral element.
• Solve on each of the 5 layers
  and

18
So, What Is the Problem?

• Symmetric, positive definite matrix
• Not a M-matrix
• Really, really huge

Many parallel solvers available
but
Memory requirements vs. solution time
19
Spectral Element

• Gauss-Lobatto discretization
• Element is the support of inner node f.e. basis
  functions

• Inner nodes
• Boundary nodes
• consisting of Edge nodes
• Vertex nodes

20
System of Equations

• Spectral element discretization
• Solve 10 times the system of equations
• Block structure
• where
• and the are
  symmetric.

21
Schur Complement-cg

• Solve Laplacian equations by Schur Complement
  Conjugate Gradient (cg)
• Preconditioner
• Take advantage of the structure of the spectral
  elements

22
Factor matrix

• Factorization of
• Schur complement
• Matrices are stored.

23
Schur Complement and Basis Transformation

• Defining the exact harmonic basis (EHB)
  transformation
• the Schur complement can be reinterpreted as
• i.e., Galerkin approach.

24
Schur Complement cg

• 1.)
• 2.)
• 3.) Solve
• 4.)

25
Schur Complement Preconditioner I

• Again, we can factor
  such that
• BUT (with j
  counter of elements/edges/)

26
Schur Complement Preconditioner II

• Replace with
• linear interpolation from vertices
  onto an edge j

27
Schur Complement Preconditioner III

• Calculate element-wise
• Approximate by
• is on edge j
  Dryja
• Derive directly
  by symbolic methods
• Bramble/Pasciak/Schatz

28
Schur Complement-pcg

• 1.)
• 2.) Solve
• 3.)
• 4.)

29
Vertex Node System

• is equivalent to a
• (non-constant) 9-point
  stencil
• Solve directly (gather on one processor)
• Combine with parallel AMG (PEBBLES)
• Special cache optimized and parallel AMG/MG for
  9-point stencil ()

30
Memory Requirements

• Laplacian in 2D
• Small example 99 elements, 5146 nodes
• M O(nelem)
• M(Schur-cg) 2.35 MB
• M(Schur-cg,pc) 2.36 MB

31
Memory requirements
99 elements, 5146 nodes
32
Simultaneous Iterations
Filtering requires in each time step
5 layers
Solve 10 Laplacian equations at once! Less
memory accesses, better cache usage
33
Improvements by Sim. Solving
2.05 faster on R10k 1.75 faster on R14k
Filtering only 15 WCPU
34
70 of Computing Time
is spent in solving
Note
35
System of Equations
Diagonal matrix
36
Schur Complement System (Pressure)
Solve via precond. cg...
37
Solving of Pressure System
Solve via pcg

• No matrices stored !
• gt Only matrix-vector
  operation available.

38
Preconditioning of Pressure System

• Original preconditioner

• New preconditioner (Lumped Schur Complement)

39
Further Improvement by Lumped Schur Complement
Speedup 1.28 on R10k 1.22 onR14k
40
Future of SEOM

• SEOM
• SEOM is currently a big, bad Fortran 90 code.
• Fortran is a dying language.
• Fortran 90 should be avoided at all costs.
• BYON

• Distributed Preprocessing Stage
• Vanishing Layer Thickness
• Non-Hydrostatic Primitive Equations
• 3D Continuous Stratification
• Integration of PEBBLES, a parallel algebraic
• multigrid solver from Linz.

41
SEOM Versions and Applications, Contd

• 3D Continuous Stratification
• density is a dependent variable
• low order SE discretization in vertical
• terrain following
• gravitational adjustment
• overflow
• flow rectification in a laboratory canyon
• test problems available on marine.rutgers.edu/po

42
Thank You