Integrating Fusion Codes Using CCA

1
Integrating Fusion Codes Using CCA an ORNL LDRD
Project

• Wael R. Elwasif Lee BerryComputer
  Science Fusion Energy
  DivisionMathematics Division
• Oak Ridge National Laboratory

2
Project Goals

• Gain experience in using CCA components for
  integrated plasma simulation.
• Explore code integration issues
• Interface Development.
• Data Coupling.
• Have model sufficiently complex to understand
  weaknesses/strengths of CCA.
• But have it fit within a two year, 1.5 FTE/year
  effort.

3
Project Members - Fusion

• Lee Berry
• Overall coordination, driver development,
  integration, design.
• Wayne Houlberg
• NTCC-like routines for calculations metrics from
  eqdsk files, advancing ion densities, and
  advancing electron and ion energy.
• Fred Jaeger
• Implementations of AORSA and routines for brining
  in equilibrium, plasma profiles into the form
  needed for AORSA

4
Project Members Computer Science Mathematics

• Wael Elwasif
• Build system, CVS, implementation strategies
  assistance, component infrastructure.
• James Kohl
• State component, Visualization development.
• David Bernhardt
• CS coordination.
• Ed DAzevedo
• Development of iterative AORSA algorithm.

5
Prototype Components

• Physics
• Equilibrium based on EFIT (F95).
• RF Heating based on AORSA (F95).
• Transport (F95).
• Driver
• Overall time stepping and coordination (Python).
• State Repository (C).
• Data couplers and transforms (F95).

6
The Physics

• Solving a model problem (nothing new vis-à-vis
  fusion).
• Couple together codes for
• Equilibrium using EQDSK files.
• Transport code from Wayne Houlberg.
• RF - AORSA2D (All Orders Spectral Algorithm)
  code from Fred Jaeger.

7
Code Status Pre-integration

• Stand alone codes.
• File-based control and input/output.
• Code-specific pre-processing (e.g. Coordinate
  translation and re-griding).
• Implemented in F95, with some language-specific
  features (e.g. optional arguments).

8
Implementation Strategy

• Getting everybody to know what CCA is, and how to
  use available tools (especially Babel/SIDL).
• Initial interface specification (keeping it
  intentionally high level).
• Getting a working component application (that
  doesnt do much).
• Repeat
• Refine interfaces and update implementation.
• Add new capabilities (e.g. remove dependency on
  input files).

9
Prototype Components Physics
10
The Physics Transport

• DATA_PATH set by driver.
• Interface evolved over project duration.
• Implementation wrapped existing F95 module (with
  one-to-one correspondence).
• Mapping between SIDL arrays and native F95 arrays
  was the most challenging aspect.

interface Xport extends gov.cca.Port int
init( in string DATA_PATH, out
string init_xp_message, out int i_flag,
) /// Advance density
equation int pushN( in double
time, in double dt, in array
ltint, 1gt k_nbc_i, )
/// Advance temperature equation int
pushT( in double time, in double
dt, in double tebc,
)
11
The Physics RF Equilibrium
interface RF extends gov.cca.Port ///
Get RF heating parameters int getRF(
in string DATA_PATH, in double time,
in double power, out string
rf_message, in int k_grid )

• DATA_PATH set by driver.
• RF
• Implementation wrapped re-factored AORSA2d core).
• Equilibrium
• Uses EQDSK file.
• Updates the equilibrium part of the state

interface Eql extends gov.cca.Port /// Get
equilibrium parameters int getEql( in
string DATA_PATH, in string file_name,
in double eq_time, out string
eq_message )
12
Prototype Components State
13
The PlasmaState Component
interface EQState extends gov.cca.Port int
initEQ( in int nx_xy, in int ny_xy,
) .. .. .. // End interface
EQStateinterface RFState extends gov.cca.Port
int initRF( in int n_rgrid, in
int n_zgrid, ) .. ..
..interface XPState extends gov.cca.Port
int initXport( in int n_r, in int
n_I ) .. .. ..

• One component implements state repository for
  Transport, Equilibrium, and RF.
• In-memory data repository.
• Allow centralization of data archiving,
  checkpointing, and restart.
• No data transform within the state component.
• Data exchange between physics components happens
  through the state, or the driver (no direct
  coupling).

14
Prototype Components Data Couplers
15
The Need for Data Couplers

• Data consumers adapt raw data to their
  specific requirements (coordinate transforms, and
  re-gridding).
• Separate this preprocessing stage into stand
  alone components.
• NOT VERY SCALABLE.
• Future goal Evolve towards generalized
  (centralized) data transforms with controllable
  mathematical properties.

16
Prototype Components Driver
17
The Driver Component

• The brains of the overall simulation.
• RAD using Python enables speedy experimentation,
  parameter tuning, and quick on-line
  visualization.
• Can be quite complex.
• Using Python allows easy access to high level
  matrix operations via the numeric python module
  whenever it makes sense.

18
Coupled Simulation It actually Works!!!
19
Implementation Issues THE BUILD

• Adopted from CCA template used in CCA tutorial.
• Was adequate for the scale and scope of this
  project
• Scaling to accommodate more components with
  varying native build systems is still an issue.
• Needed tweaking when deployed to different Linux
  platforms (compilers, location of legacy
  libraries).
• Legacy libraries build system changed as well.

20
Implementation Issues CVS

• Initially used only for the components.
• Extended to include a snapshot of the legacy
  codes.
• Legacy codes re-factored (change into libraries,
  streamline file access).
• Question How to manage continuous parallel
  development of legacy codes and integration?

21
Lessons Learned

• Throwing over the fence does NOT work.
• Interface development was easy We control
  everything.
• The build system is VERY important.
• Language interoperability using Babel makes
  things possible, but not necessarily easy.
• Incorporating independently developed F95 codes
  into a single application has its own pitfalls
  (e.g. compiler flags for code correctness).
• Integration exercises codes in unforeseen ways
  (bugs found in established codes).

22
Future Plans

• Next four months.
• Implement iterative RF module with time varying
  densities.
• Use HDF5 as back-end for the state component.
• Run on the Altix at ORNL.