Support for Adaptive Computations Applied to Simulation of Fluids in Biological Systems

1
Support for Adaptive Computations Applied to
Simulation of Fluids in Biological Systems

• Immersed Boundary Method Simulation in Titanium

2
Objectives

• Provide easy-to-use, high-performance tool for
  simulation of fluid flow in biological systems.
• Demonstrate the Titanium compiler and language.
• Allow heart simulation on large-scale parallel
  machines.

3
Outline

• Immersed Boundary Method
• Titanium
• Immersed Boundary Method in Titanium

4
Immersed Boundary Method

• Developed at New York University by Peskin
  McQueen to model biological systems where elastic
  fibers are immersed in an incompressible fluid.
• Mammalian heart, blood platelets, sea urchin
  embryos
• Fibers (e.g., heart muscles) modeled by list of
  fiber points
• Fluid space modeled by a regular lattice

5
Immersed Boundary Method Structure

• 4 steps in each timestep

Fiber activation force calculation
Fiber Points
Interpolate Velocity
Spread Force
Interaction
Fluid Lattice
Navier-Stokes Solver
6
IB Method Steps 1 and 2

• Fiber Activation Force calculation
• Application-specific
• For the heart, use an elastic spring law
• Spread Force
• Spread forces from fiber points list to fluid
  lattice via the Dirac Delta function.

7
IB Method Steps 3 and 4

• Navier-Stokes Solver
• Calculate fluid velocities
• Uses a 3D FFT
• Interpolate velocity
• Gather velocities of fiber points from fluid
  lattice via Dirac delta function
• Move the fiber points

8
Challenges to Parallelization

• Irregular fiber lists need to interact with
  regular fluid lattice.
• Trade-off between load balancing of fibers and
  minimizing communication
• Efficient scatter-gather across processors
• Need a scalable elliptic solver
• Plan to uses multigrid
• Eventually addAdaptive Mesh Refinement
• New algorithms under development at LBNL

9
Heart Application of the IB Method

• Heart simulation used to design artificial heart
  valves

10
Outline

• Immersed Boundary Method
• Titanium
• Immersed Boundary Method on Titanium

11
Titanium Motivation

• Applications are increasingly complex
• Want classes, overloading, linked data
  structures
• C is hard to read, modify and tune
• Machines are increasingly complex
• Want compiler help for optimizations
• Want clear performance model and programmer
  control
• Java is a better C
• Safe strongly typed, garbage collected
• Performance is poor due to

12
Titanium for Scientific Computing

• Java dialect for high performance
• Added constructs for performance expressiveness
• Immutable, value classes
• SPMD parallelism with a global address space
• Multidimensional arrays
• Templates
• Region-based memory management
• Compiled to C (no JVM) with lightweight messaging
  (Active Messages, LAPI, shmem)

13
SPMD Parallelism in Titanium

• Explicitly parallel model
• Fixed number of threads at program startup
• Usually one thread per processor
• Global address space
• Processors can access remote data by reading and
  writing through global references (pointers)
• Bulk communication happens when copying arrays
• Compiler automatically converts global pointers
  local ones (up to 2x speedup for LQI)
• Compiler detects synchronizations bugs in barriers

14
Value Classes in Titanium

• Java has two distinct kinds of values
• Primitive scalar types boolean, double, int,
  etc.
• Objects user-defined and library types
• implicit level of indirection (pointer to)
• Titanium adds support for small objects
• Look like classes with immutable keyword
• Stored in place and passed by coping
• Examples
• Complex type
• Points used to index Titanium arrays

15
Titanium Arrays

• Java arrays are 1-dimensional
• Arrays of arrays are inefficient
• Titanium adds multidimensional arrays
• Indexed by Points (tuples of ints)
• Algebra over Domains (sets of points)
• Helps with hierarchical algorithms, e.g.,
  multigrid
• One array may be a subarray of another
• e.g., a is interior of b, or a is all even
  elements of b
• Foreach loops help compiler optimize arrays
• Within 2x of C for multigrid kernels
• Bulk I/O provided on arrays (2x-40x speedup!)

16
Titanium with Other Languages

• Native methods are sometimes useful
• Performance allows for comparisons with other
  compilers and additional control
• Libraries have interfaced to other systems like
  PetSC and ParMetis
• Requires understanding of underlying Titanium
  implementation in C.
• Lower entry cost than Java the native method is
  simply included into the generated code

17
Titanium Implementation

• Run time system and compiler for
• Uniprocessors
• SMP running POSIX threads
• Clusters with
• Shared memory - SGI Origin cluster (ANL), Tera
  MTA
• Global Address Space - T3E (NERSC)
• Active Messages - NOW Millennium (UCB)
• LAPI - IBM SP2, SP3 (SDSC)

18
Outline

• Immersed Boundary Method
• Titanium
• Immersed Boundary Method on Titanium

19
Immersed Boundary Generic Software

• Written by Cowen at NYU
• Implements subset of the IB method adequate for
  the heart
• Runs on vector machines with shared memory

20
Immersed Boundary on Titanium

• IBGS rewritten in Titanium.
• Running since October
• Contractile torus
• runs on Berkeley NOW and SGI Origin
• Needed for heart
• Input file format
• Performance tuning
• Uniprocessor (C code used temporarily in 2
  kernels)
• Communication

21
Contractile Torus Visualization
22
Immersed Boundary on Titanium

• Performance Breakdown (torus simulation)

23
Immersed Boundary on Titanium
24
Future work

• Improve performance
• Especially on SP machines (Blue Horizon)
• Add functionality
• Bending angles, anchorage points, source sinks)
  to the software package.
• Add adaptability to NS solver (AMR)
• Needed for scaling and more accurate modeling of
  fluid features in heart