Support for Adaptive Computations Applied to Simulation of Fluids in Biological Systems

1
Support for Adaptive Computations Applied to
Simulation of Fluids in Biological Systems

• Kathy Yelick
• U.C. Berkeley

2
Project Summary

• Provide easy-to-use, high performance tool for
  simulation of fluid flow in biological systems.
• Using the Immersed Boundary Method
• Enable simulations on large-scale parallel
  machines.
• Distributed memory machine including SMP clusters
• Using Titanium, ADR, and KeLP with AMR
• Specific demonstration problem Simulation of the
  heart model on Blue Horizon.

3
Outline

• Short term goals and plans
• Technical status of project
• Immersed Boundary Method
• Software Tools
• Solvers
• Next Steps

4
Short Term Goals for October 2001

• IB Method written in Titanium (IBT)
• IBT Simulation on distributed memory
• Heart model input and visualization support in
  IBT
• Titanium running on Blue Horizon
• IBT users on BH and other SPs
• Performance tuning of code to exceed T90
  performance
• Replace solver with (adaptive) multigrid

5
IB Method Users

• Peskin and McQueen at NYU
• Heart model, including valve design
• At Washington
• Insect flight
• Fauchy et al at Tulane
• Small animal swimming
• Peter Kramer at RPI
• Brownian motion in the IBM
• John Stocky at Simon Fraser
• Paper making
• Others
• parachutes, flags, flagellates, robot insects

6
Building a User Community

• Many users of the IB Method
• Lots of concern over lack of distributed memory
  implementation
• Once IBT is more robust and efficient (May 01),
  advertise to users
• Identify 1 or 2 early adopters
• Longer term workshop or short course

7
Long Term Software Release Model

• Titanium
• Working with UPC and possibly others on common
  runtime layer
• Compiler is relatively stable but needs ongoing
  support
• IB Method
• Release Titanium source code
• Parameterized black box for IB Method with
  possible cross-language support
• Visualization software is tied to SGI

8
Immersed Boundary Method

• Developed at NYU by Peskin McQueen to model
  biological systems where elastic fibers are
  immersed in an incompressible fluid.
• Fibers (e.g., heart muscles) modeled by list of
  fiber points
• Fluid space modeled by a regular lattice

9
Immersed Boundary Method Structure

• 4 steps in each timestep

Fiber activation force calculation
Fiber Points
Interpolate Velocity
Spread Force
Interaction
Navier-Stokes Solver
Fluid Lattice
10
Challenges to Parallelization

• Irregular fiber lists need to interact with
  regular fluid lattice.
• Trade-off between load balancing of fibers and
  minimizing communication
• Efficient scatter-gather across processors
• Need a scalable elliptic solver
• Plan to uses multigrid
• Eventually add Adaptive Mesh Refinement
• New algorithms under development by Colellas
  group

11
Tools used for Implementation

• Titanium supports
• Classes, linked data structures, overloading
• Distributed data structures (global address
  space)
• Useful for planned adaptive hierarchical
  structures
• ADR provides
• Help with analysis and organization of output
• Especially for hierarchical data
• KeLP provides
• Alternative programming model for solvers
• ADR and KeLP are not critical for first-year

12
Titanium Status

• Titanium runs on uniprocessors, SMPs, and
  distributed memory with a single programming
  model
• It has run on Blue Horizon
• Issues related to communication balance
• Revamped backends are more organized, but BH
  backend not working right now
• Need to replace personnel

13
Solver Status

• Current solver is based on 3D FFT
• Multigrid might be more scalable
• Multigrid with adaptive meshes might be more so
• Balls and Colella algorithm could also be used
• KeLP implementations of solvers included
• Note McQueen is looking into solver issues for
  numerical reasons unrelated to scaling
• Not critical for first year goals

14
IB Titanium Status

• IB (Generic) rewritten in Titanium.
• Running since October
• Contractile torus
• runs on Berkeley NOW and SGI Origin
• Needed for heart
• Input file format
• Performance tuning
• Uniprocessor (C code used temporarily in 2
  kernels)
• Communication

15
Immersed Boundary on Titanium

• Performance Breakdown (torus simulation)

16
Immersed Boundary on Titanium
17
Next Steps

• Improve performance of IBT
• Generate heart input for IBT
• Recover Titanium on BH
• Identify early user(s) of IBT
• Improve NS solver
• Add functionality
• Bending angles, anchorage points, source sinks)
  to the software package.

18
Adaptive Computations for Fluids in
Biological Systems
Immersed Boundary Method Applications
Human Heart (NYU)
Embryo Growth (UCB)

• Yelick(UCB), Peskin (NYU), Colella (LBNL),
  Baden (UCSD), Saltz (Maryland)

Blood Clotting (Utah)
Robot Insect Flight (NYU)
Pulp Fibers (Waterloo)
Heart (Titanium)
Insect Wings
Flagellate Swimming

Application Models
Generic Immersed Boundary Method (Titanium)
Extensible Simulation
Spectral (Titanium)
Multigrid (KeLP)
AMR
Solvers
19
General Questions

• - How has your project addressed the goals of the
  PACI program (providingaccess to tradition HPC,
  providing early access to experimental
  systems,fostering interdisciplinary research,
  contributing to intellectualdevelopment,
  broadening the base)?- What infrastructure
  products (e.g., software, algorithms, etc.) have
  you produced?- Where have you deployed them (on
  NPACI systems, other systems)?- What have you
  done to communicate the availability of
  thisinfrastructure?- What training have you
  done?- What kind/size of community is using your
  infrastructure?- How have you integrated your
  work with EOT activities?- What scientific
  accomplishments - or other measurable impacts
  notcovered by answers to previous questions -
  have resulted from its use?- What are the
  emerging trends/technologies that NPACI should
  buildon/leverage?- How can we increase the
  impact of NPACI development to date?- How can we
  increase the community that uses the
  infrastructure you'vedeveloped?