Inner Workings And Architecture of FLASH

1
Inner Workings And Architecture of FLASH

• Anshu Dubey Katie Antypas
• June 4, 2006

2
Overview

• What is FLASH
• Basics of FLASH Architecture
• Similarities and differences between versions 2
  and 3
• Units
• Inheritance
• Naming
• Basics behind a problem setup
• Enough of an idea of these concepts to be able to
  look at a sample setup and understand what is
  happening.

3
The Flash Code
Shortly Relativistic accretion onto NS
Flame-vortex interactions
Compressed turbulence
Type Ia Supernova

• The Flash code
• Parallel, adaptive-mesh simulation code
• Designed for compressible reactive flows
• Has a modern CS-influenced architecture
• Can solve a broad range of (astro)physics
  problems
• Portable- runs on many massively-parallel systems
• Scales and performs well
• Is available on the web http//flash.uchicago.edu

Gravitational collapse/Jeans instability
Wave breaking on white dwarfs
Intracluster interactions
Laser-driven shock instabilities
Nova outbursts on white dwarfs
Rayleigh-Taylor instability
Orzag/Tang MHD vortex
Helium burning on neutron stars
Cellular detonation
Magnetic Rayleigh-Taylor
Richtmyer-Meshkov instability
4
What FLASH Provides

• Physics
• Hydrodynamics
• PPM
• MHD
• Relativistic PPM
• Equation of State
• Ideal gas
• Multigamma
• Helmholtz
• Nuclear Physics
• Gravity
• Cosmology
• Particles

• Infrastructure
• Setup
• Mesh Management
• AMR Paramesh
• UG
• Parallel I/O
• hdf5, pnetcdf
• Monitoring
• Runtime and post-processing visualization
• Regular testing toolkit
• Unit test framework

5
A Little FLASH History
BAM

• FLASH0
• Paramesh2, Prometheus and EOS/Burn
• FLASH1
• Smoothing out the smash
• First form of module architecture inheritance
• FLASH2
• Untangle modules from each other (Grid)
• dBase
• Concept of levels of users
• FLASH3
• Stricter interface control unit architecture
• Taming the database
• Better encapsulation and isolation
• Community supported and developed code

6
FLASH Audiences
FLASH Application
Developer
End User
Application Programmer

• Works on just about everything
• Grid development
• Data access
• Architecture

• Initial conditions
• Boundary conditions
• Runtime parameters
• Select units, and their specific implementations
• Implement functions specific to an application,
  if they are not provided by FLASH

• Develop physics Unit
• Publish API
• Use other units API

7
FLASH Code Basics

• An application code, composed of units/modules.
  Particular modules are set up together to run
  different physics problems.
• Performance, Testing, Usability, Portability
• Fortran, C, Python,
• More than 560,000 lines of code
• 75 code, 25 comment
• Very portable
• Scaling to tens of thousand procs

Internal Release
8
FLASH on BG/L - Sod weak scaling
Hydro Time Constant Work Per Processor using AMR
Total Time
Number of Processors
Source K.M. Riley, Argonne National Lab
9
Hydro and Particles Scaling
10
Basic Computational Unit Block

• The grid is composed of blocks
• Most often all blocks have same dimensions
• Cover different fraction of the physical domain.
• In AMR blocks at different levels of refinement
  have different grid spacing.

11
FLASH Architecture
Four Cornerstones
Setup tool assemble simulation
Config files Tell setup how to Assemble simulation
Unit Architecture API Inheritance Data management
Driver Organize interactions Between units
12
Whats a FLASH Unit?

• FLASH basic architecture unit
• Component of the FLASH code providing a
  particular functionality
• Different combinations of units are used for
  particular problem setups
• Publishes a public interface for other units
  use.
• Ex Driver, Grid, Hydro, IO etc
• Fake inheritance by use of directory structure
• Interaction between units governed by the Driver
• Not all units are included in all applications

13
FLASH Units
Infrastructure
I/O
Runtime Params
Grid
monitoring
Physics
Profiling
Hydro
MHD
Driver
Burn
Gravity
Logfile
Simulation
14
Unit Architecture
Grid
Data Module

• Top level
• API
• Unit Test

block data time step etc
Wrapper
Driver
Kernel
15
Unit Hierarchy
Unit API/stubs
UnitMain Common API implementation
UnitSomething API implementation
common More common impl
Impl_1 Remaining API impl
kernel
Impl_2 Remaining API impl
Impl_3 Remaining API impl
kernel
kernel
kernel
16
Inheritance Through Directories Eos
Stub Implementations of the three functions at
the top level
Eos_ init
Eos
Eos_ wrapped
There is only one subunit
EosMain
Eos/EosMain Replaces the stub with
an implementation common to all formulations of
EOS
Eos_wrapped
Gamma
Specific implementation
Eos/EosMain/Gamma implements gamma versions Of
Eos_init and Eos
Eos_init
Eos
Another implementation, which will have its own
Eos and Eos_init etc.
Multigamma
17
The Config File

• Declare solution variables, fluxes
• Declare runtime parameters
• Sets defaults
• Lists required, requested or exclusive units
• Config files are additive down the directory tree
  - no replacements

18
FLASH Setup Script Implements Architecture

• Python code links together needed physics and
  tools for a problem
• Parses Config files to
• Determine a self consistent set of units to
  include
• If a unit has multiple implementations, finds out
  which implementation to include
• get list of parameters from units
• Determines solution data storage
• Creates a file defining global constants set at
  build time
• Builds infrastructure for mapping runtime
  parameters to constants as needed
• Configures Makefiles properly

19
Pulling it All Together

• Choose a problem simulation
• Run setup to configure that problem
• Everything is in a new top-level directory
• object
• Make
• Run
• flash.par for runtime parameters
• Defaults already set from particular units

20
Adding a new Simulation

• A basic problem setup
• Config file
• Required physics modules
• flash.par
• Default list runtime parameter configuration
• Simulation_initBlock
• Initial conditions for the problem set block by
  block
• Many other possible files
• Driver, Refinement algorithms, User defined
  boundary conditions
• Any files in setup take precedence

More about setting up your own problem in the
hands on session later this afternoon
21
Provided Driver

• Provided Second order, state form, strang split

FLASH3
FLASH2
flash.F90
Flash.F90
Initialize Loop over timesteps evolve
adjust dt output visualize End loop Finalize
Driver_initFlash Driver_evolveFlash Driver_finaliz
eFlash
Driver Unit
Driver_evolveFlash.F90
Loop over timesteps set time step call
physics set time step repeat physics
Grid_updateRefinement adjust dt output End
loop
evolve.F90
set time step do physics set time step repeat
physics Mesh_updateGrid
22
Flash Grids Paramesh

• FLASH original design was Paramesh specific
• Block Structured
• All blocks have same dimensions
• Blocks at different refinement levels have
  different grid spacings and thus cover different
  fractions of the physical domain
• Fixed sized blocks specified at compile time
• Every section of the code assumed knowledge of
  block size
• Good for capturing shocks
• Currently removing fixed block size requirement
  from code to open door for other mesh packages
  like a uniform grid, squishy grid and patched
  base grid

In choosing Paramesh, the original FLASH code
architects chose simplicity of the Paramesh
structure over a patch based mesh.
23
Infrastructure Mesh Packages in Flash
Paramesh
Uniform Grid

• one block per proc
• No AMR related overhead

• Block Structured
• Fixed sized blocks
• Specified at compile time
• Not more than one level jump at fine coarse
  boundaries

24
Building and Running an Application
Runtime Inputs
Driver
Grid
mesh
Simulation/ setup
I/O
Physics
Profiler
25
IO/Viz Outline

• I/O Challenges with Scientific Computing
• speed, portability, file size
• serial, parallel multi-file, MPI-IO
• I/O Libraries
• hdf5
• parallel-netcdf
• FLASH I/O specifics
• parameter input file
• checkpointing and restarts
• plotfiles
• Visualization
• runtime visualization
• fidlr3
• flashView

26
I/O Challenges Portability
Taking a basic binary output file from one
machine and trying to use it on another often
causes problems

• Portability
• big endian, little endian
• size of types vary
• data alignment in memory

4 byte int
2 byte int
int
int
int
double
24
0
4
8
12
16
20
27
I/O challenges File size and Speed

• File size
• Files are huge --in FLASH a typical checkpoint
  file typically 5 GB and up to 50 GB for a
  research simulation (plotfiles 2GB)
• Need access to large storage areas
• Data transfer takes lots of time (.5 MB/sec from
  labs)
• I/O can be a major performance bottle neck
• slow file system
• writing to disk takes time

It isnt hard to have speed, portability or
usability. It is hard to have speed, portability
and usability in the same implementation
28
Serial I/O
0
1
2
3
4
5
processors

• Each processor sends its data to the master who
  then writes the data to a file
• Advantages
• Dont need a parallel file system
• Simple
• Disadvantages
• Not scalable
• Not Efficient

File
29
Parallel I/O Multi-file
0
1
2
3
4
5
processors
File
File
File
File
File
File

• Each processor writes its own data to a separate
  file
• Advantages
• Fast!
• Disadvantages
• can quickly accumulate many files
• hard to manage
• requires post processing

30
Parallel I/O Single-file
1
2
3
4
5
0
processors
File

• Each processor writes its own data to the same
  file using MPI-IO mapping
• Advantages
• single file
• scalable (to how many procs is debatable...)
• Disadvantages
• requires MPI-IO mapping or other higher level
  libraries

31
Parallel I/O single file
0
1
2
3
4
5
processors
array of data
Each processor writes to a section of a data
array. Each must know its offset from the
beginning of the array and the number of elements
to write
32
I/O Libraries

• FLASH works with 2 different I/O libraries
• HDF5
• Parallel-NetCDF
• Use MPI-IO mappings
• Both Portable libraries
• Both can translate data between systems
• Scientific Data mostly stored in multidimensional
  arrays
• FLASH3 also supports a basic direct FORTRAN I/O
  -- use only as a last resort!

33
FLASH I/O

• FLASH can use either HDF5 or Parallel-NetCDF for
  I/O
• Initially very optimistic about PnetCDF because
  it appeared to be twice as fast as HDF5

Source J.B. Gallagher on ASC White
34
Parameter Input File - flash.par

• To change parameters in a simulation edit the
  flash.par file
• Defaults are set in Config files and any value in
  the flash.par overides the Config values
• Parameter MUST be declared in Config file!!!
• Controls parameters for all units (examples
  below)
• Grid parameters - lrefine_max, lrefine_min,
  geometry
• Physics parameters - flame_speed
• I/O parameters - restart, checkpointFileNumber

35
Flash Checkpoint (restart) Files

• Checkpoint file -- a file that saves all the
  data you need to restart your simulation from the
  exact state it left off
• Grid Data
• refinement level
• nodetype (parent, child)
• bound box
• coordinates
• unknown variables (density, pressure, temperature
  etc)
• Simulation Data
• time
• dt
• timestep
• MetaData
• time stamp
• build directory
• setup problem etc

For any number of reasons your simulation could
be interrupted - the machine crashes, the queue
window closes, the machine runs out of disk space
etc. Rather than starting again from the
beginning of your simulation you can restart your
problem using a flash checkpoint files
36
Checkpoint Parameters

• There are a number of ways to control the
  occurrence and spacing of checkpoint dumps (names
  have been clarified for FLASH3)
• timesteps between dumps - (parameter
  checkpointFileIntervalStep)
• simulation time seconds between dumps -
  (parameter checkpointFileIntervalTime)
• wall clock seconds between dumps - (parameter
  wall_clock_checkpoint)
• very useful when running in a queue or when only
  a certain amount of time is available on a
  machine
• creating a file called .dump_checkpoint will
  force a checkpoint file to be dumped immediately
• creating a .dump_restart file will force a
  checkpoint and halt the code
• rolling checkpoint ability - (parameter
  rolling_checkpoint)
• checkpoint files are big! 5 - 50GB for a large
  science run
• this feature allows you to keep only a specified
  number of checkpoints to save disk space

The FLASH code support website gives detailed
explanations of all runtime parameters
37
Restarting a Simulation
To restart a simulation, in your flash.par file
set the runtime parameter restart .true.
the checkpointFileNumber parameter to the
desired checkpoint number

• Because the checkpoint files save the exact state
  of the simulation you can restart your simulation
  from any FLASH checkpoint file
• Can start with Parallel IO if previous run was
  written in serial
• Can restart with a different number of processors
  than the previous run

38
Restarting on a different number of processors
0
1
2
3
4
5
processors
array of data
restart
0
1
2
processors
array of data
39
Flash Plotfiles and Particle Files
Plotfiles and particle files output data for
visualization and analysis purposes

• Plotfiles are smaller than checkpoints
• Only output variables needed for visualization or
  analysis
• Data stored in single precision
• Control plotfile frequency with the runtime
  parameters
• plotfileNumber
• plotfileIntervalStep
• plotfileIntervalTime
• Particle files stored in single precision
• Control particle file frequency with the runtime
  parameters
• particleFileNumber
• particleFileIntervalStep
• particleFileIntervalTime

40
Visualization
41
Visualization

• Runtime Visualization
• Fidlr3
• FlashView

42
Runtime Visualization

• Runtime Visualization module
• not automatically included -- can add it into
  your Modules file to set up problem with Runtime
  Viz
• Allows user a quick and dirty view of the
  simulation
• Only requires the widely available png library
• Creates png files in current directory during
  runtime

43
Fidlr3 (FLASH IDL Routines)
Fidlr is an application written in idl to
visualize Flash checkpoint and plotfiles

• IDL tool for 2d visualization
• Available with FLASH directly
• Can step through plots to view sequences or see
  movies
• Can visualize 3d data by looking at slices
• Make histograms
• Additionally, if you have access to idl and want
  to do more analysis you can use the routines to
  read in data, statistical analysis, max min vaules

44
Example of 3d slice taken by FILDR3
45
FlashView
FlashView is a 3d visualization package developed
for FLASH data at Argonne National Lab by Mike
Papka and Randy Hudson

• Features
• Reads native FLASH plotfiles
• XYZ cut planes features so entire range of data
  for a variable can be projected on a plane
• Renders 3d iso-surfaces of Flash Data
• Fully parallel (both reading the file and
  calculations)
• Parallelism allows for the ability to look at
  very large data sets interactively
• Displays mulitple (up to 5) variables at the same
  time
• various color maps, zoom, rotation, translation
  etc
• shows grid structure
• movies making
• Currently only supports hdf5 format

Yes ... we think it is difficult to install
too... Key is to install exact version of
recommended packages Ask Dan if you have
questions
46
FlashView
47
Example of an isosurface (single variable)
48
Another example with two variables
49
Another example/ isosurface with cut plane
50
Isosurface with Grid
51
Issues with Cell-centered vs. Corner Interpolated
Data
52
Issues with cell-centered vs. corner interpolated
data