The Paramesh Grid Implementation in FLASH3 Klaus Weide May 2006

1
The Paramesh Grid Implementation in
FLASH3Klaus WeideMay 2006
2
Preliminaries

• Scope
• The PARAMESH3 (actually now PARAMESH 4.0)
  implementation of the Grid unit in FLASH3
• Ignoring particles (a related but separate topic)
• Ignoring NO_PERMANENT_GUARDCELLS etc.
• What users of FLASH3 need to know about this
  Grid.
• In particular, writers of non-infrastructure
  code that will work with FLASH3.

3
Preliminaries

• A Warning that Shall Be Repeated
• This presentation includes discussion of internal
  structures of the Grid implementation, in order
  to explain the function of the Grid.
• Please do not access these structures in code
  outside the Grid unit. You may get away with it
  for a while, but we don't want to support it!
• If you break it, you get to keep both parts.
• Please do use the official interfaces.
  For example, use Grid_getBlkIndexLimits
  instead of constants NXB, NYB, etc.
• This will keep your code working if the Grid
  implementation changes!

4
Basic Computational Unit Block

• The grid is composed of blocks
• Blocks are composed of interior cells and
  surrounding guard cells
• Paramesh All blocks have the same size (number
  of cells), can have different resolution
• PM in Flash3 Block size is determined at setup
  time.

5
First Look at Paramesh Grid

• Purpose of the Grid represent data
• more on UNK variables etc. below
• Each block resides on exactly one processor (at a
  given point in time)
• Limitations imposed by Paramesh
• Same number of cells, guard cell layers
• Resolution (Delta) of a block changes by
  multiples of 2
• Resolution of neighbors differs at most by factor
  of 2

6
How Blocks are Identified

• At a given time, a block is globally uniquely
  identified by a pair (PE, BlockID), where
• 0 lt PE lt numprocs
• 1 lt BlockID lt MAXBLOCKS
• Locally, BlockID is sufficient to specify a block
• User code can't directly access remote blocks
  anyway
• Morton Numbers provide another way to identify
  blocks globally (more later)

7
How Blocks are Stored

• Solution data,
• per-block meta data,
• tree information (for local blocks!)
• are stored in F90 arrays declared like this
• real, dimension(,,,,MAXBLOCKS) UNK
• real, dimension(,MAXBLOCKS) bnd_box
• integer, dimension(,MAXBLOCKS) parent
• MAXBLOCKS is a hardwired constant (from setup
  time)
• Inactive (non-leaf) blocks also use storage
• These structures are internal to the Grid unit
  and should not be accessed directly by other
  code.
• Use the appropriate Grid_something subroutine
  calls instead!

8
How Blocks are Woven Together I

• The Tree (1d binary tree 3d oct tree)
• parent/child relationships
• Neighbors
• geometric relationships
• Morton Ordering
• a way to serialize blocks
• used only internally, implicitly
• Determines storage of blocks

9
Connecting Blocks The Tree

• The Tree (1d binary tree 3d oct tree)
• parent/child relationships
• three types of blocks
• LEAF physics acts only on data of these blocks
• PARENT have at least one LEAF child
• ANCESTOR (inactive) other, data may be invalid
• Not all blocks have a parent there is at least
  one root node, there may be many!
• Each block has a refinement level,
  1 lrefine(blockID)
  lrefine_max

10
Connecting Blocks Neighbors

• Neighbor relationships
• neigh array neighbors across faces
• surr_blks surrounding blocks incl. diagonal
• regenerated by PARAMESH3 from other information
• wrap around at periodic boundaries
• Combined Tree and Neighbor info
• gr_gid array
• combines parent/child/neigh info
• This is how linkage info is stored in checkpoint
  files

11
Connecting Blocks Morton Order

• Morton Ordering
• a way to serialize blocks
• Morton Function maps blocks to integers,
  'lt' relation between integers gt morton order of
  blocks
• determines
• how blocks are ordered in memory in each
  processor AND
• how they are distributed across processors
• Morton Curve an illustration of the Morton order
• connect centers of leaf blocks in M. order

12
How Blocks are Woven Together II

• Warnings
• The structures linking blocks should not be
  accessed directly by user code
• parent, neigh, etc. may point to remote blocks
• The implementation of Morton function is internal
  magic of Paramesh. You don't need to understand
  it to use FLASH3/Paramesh.
• User code generally does not need to know where a
  block's neighbor is. It can rely on Paramesh to
  fill guard cells from neighbors correctly.

13
Per-Block Meta-Information

• Paramesh keeps information associated with each
  local block
• parent, child, surr_blks, neigh block linkage
• lrefine, nodetype tree-related information
• coord, bnd_box, bsize physical coordinates
• There are public accessor interfaces
  Grid_getThisOrThat to get the parts of this
  information that may be useful to user code. Use
  them!

14
Block Data Fluid Variables I

• FLASH3 provides the following kinds of variables
• UNK variables stored in adjacent slots in UNK
  array for each cell
• cell-centered (or 'volume-centered')
• basic hydrodynamic and other physical continuous
  variables, aka. solution variables, aka.
  "unknowns".
• data maintained by PARAMESH
• UNK contains three kinds of variables, from
  different Config declarations
• General solution variables
• VARIABLE name e.g., VARIABLE dens -gt DENS_VAR
• Species mass fractions
• SPECIES NAME e.g., SPECIES NI56
• For technical reasons there is always at least
  MFRAC_SPEC
• "mass scalars" - additional passiveley advected
  variables
• MASS_SCALAR NAME e.g., MASS_SCALAR YE

15
Block Data Fluid Variables II

• Additional kinds of variables
• GRIDVAR aka. SCRATCH variables
• GRIDVAR name e.g., GRIDVAR otmp
• cell-centered (or face-centered)
• for temporary use, data is not meant to survive
  Grid changes
• data maintained by FLASH3
• Face variables
• FACEVAR name e.g., FACEVAR Mag
• face-centered
• currently only used by MHD
• data maintained by PARAMESH
• work space mentioned for completeness
• maintained by PARAMESH, not currently used by
  FLASH3
• Fluxes mentioned for completeness
• FLUX name e.g., FLUX rho
• maintained by PARAMESH/FLASH3, used in Hydro

16
Applying Boundary Conditions

• Specify type of Boundaries in flash.par
• Inner boundaries are now also possible
• Initialization from checkpoint file if restart is
  TRUE
• Non-default boundary condition handling
• physics units access and modify block data
• Usually, provide custom Grid_applyBCEdge
• provide custom Grid_applyBCEdgeAllUnkVars when
  needed
• Not recommended provide custom amr_1blk_bcset
• NoDriver unit call Grid_updateRefinement after
  each time step
• Hydrostatic boundary conditions
• work in progress
• Implementation as in FLASH2 default is now
  available

17
Life of The Mesh I

• What happens to the Grid, from start to finish of
  a calculation
• Initialization
• Initialization from scratch if restart is FALSE
• Initialization from checkpoint file if restart is
  TRUE
• Changes during Evolution
• physics units access and modify block data
• physics units call Grid_fillGuardCells when they
  need it
• Driver unit call Grid_updateRefinement after each
  time step
• Data and meta info are periodically stored in
  checkpoint files.

18
Life of The Mesh Initialization from scratch

• How is the Grid constructed initially?
• One or more initial blocks are created, to become
  root block(s) of tree(s) see runtime parameters
  nblockx, nblocky, nblockz
• These are distributed by FLASH3 across processors
  from the very beginning (parallel divide
  domain), as evenly as possible
• Some of the initial blocks can now be turned into
  internal boundary blocks (obstacles), see
  Simulation_defineDomain
• PARAMESH magic machinery is let loose on the
  remaining initial block(s) repeatedly until
  desired lrefine_max can be reached. It takes care
  of
• creating new (finer) blocks as children of
  coarser parents
• redistributing blocks across processors
  (rebalancing)
• User can influence by
• setting runtime parameters (lrefine_max,
  refine_var_n, etc.)
• supplying custom Grid_markRefineDerefine
  implementation

19
Life of The Mesh Initialization from checkpoint

• How is the Grid reconstructed ?
• Checkpoint contains Grid blocks in original (PE,
  blockID) order (Morton Order)
• This logically one-dimensional sequence of blocks
  is chopped up and distributed evenly across
  available processors.
• The same applies to meta info (e.g., gr_gid for
  tree info)
• Each processor reads and stores only the chunk
  assigned to it. No processor stores all the
  global tree data.
• PARAMESH3 is called to initialize additional data
  structures.

20
Life of The Mesh Evolution

• How does the Grid change during a simulation ?
• Physics units access and modify block data
• Grid_fillGuardCells is called when needed
• PARAMESH takes care of communication
• interpolation and restriction of data may occur
• Driver unit calls Grid_updateRefinement
  periodically
• PARAMESH magic machinery is let loose on the
  blocks. It takes care of
• creating new (finer) blocks as children of
  coarser parents
• deleting leaf blocks where reolution is too high
• redistributing blocks across processors
  (rebalancing)
• User can influence by
• setting runtime parameters (lrefine_max,
  refine_var_n, etc.)
• supplying custom Grid_markRefineDerefine
  implementation

21
Life of The Mesh The End

• How does the Grid save data ?
• The state of the blocks and meta-info is saved to
  checkpoint files
• Blocks end up stored sequentially in global
  (PE,blockID) order, i.e., Morton
  Order.
• Some more on this in IO part of tutorial.