Gyrokinetic particle-in-cell simulations of plasma microturbulence on advanced computing platforms

1
Gyrokinetic particle-in-cell simulations of
plasma microturbulence on advancedcomputing
platforms

• Stephane Ethier
• Princeton Plasma Physics Laboratory
• SCIDAC 2005 Conference
• San Francisco, CA

Work performed under the DOE SCIDAC Center for
Gyrokinetic Particle Simulation of Turbulent
Transport in Burning Plasmas
2
The Ultimate Burning Plasma
Fusion Powers the Sun and Stars
Can we harness Fusion power on Earth?
3
The Case for Fusion Energy

• Worldwide demand for energy continues to increase
• Due to population increases and economic
  development
• Worldwide oil and gas production is near or past
  peak
• Need for alternative source coal, fission,
  fusion
• Increasing evidence that release of greenhouse
  gases is causing global climate change
• This makes nuclear (fission or fusion) preferable
  to fossil (coal)
• Fusion has clear advantages over fission
• Inherently safe (no China syndrome)
• No weapons proliferation considerations
  (security)
• Greatly reduced waste disposal problems (no Yucca
  mountain)
• Can produce electricity and hydrogen
• Abundant fuel, available to all nations
• Deuterium and lithium supply will last 1000s of
  years

4
Fusion Reaction

• The two ions need to break the Coulomb barrier to
  get close enough to fuse together

5
(No Transcript)
6
Putting the Sun in a Bottle
7
The Most Successful Magnetic Confinement
Configuration is the Tokamak
plasma
magnets
magnetic field
8
Fusion Experiments in the World
9
We know we can make Fusion Energy The Challenge
now is to make it Practical!
J. B. Lister
10
Fusion DOE OFES 1 Item on List of Priorities
November 10, 2003Energy Secretary Spencer
Abraham Announces Department of Energy20-Year
Science Facility PlanSets Priorities for 28 New,
Major Science Research Facilities 1 on the list
of priorities is ITER, an unprecedented
international collaboration on the next major
step for the development of fusion 2 is
UltraScale Scientific Computing Capability
11
Plasma Science Challenges

• Macroscopic Stability
• What limits the pressure in plasmas?
• Wave-particle Interactions
• How do particles and plasma waves interact?
• Microturbulence Transport
• What causes plasma transport?
• Plasma-material Interactions
• How can high-temperature plasma and material
  surfaces co-exist?

12
Challenge to Theory Simulations
Huge range of spatial and temporal scales
Overlap in scales often means strong (simplified)
ordering not possible
13
Major Fusion Codes
14
Importance of Turbulence inFusion Plasmas

• Turbulence is believed to be the mechanism for
  cross-field transport in magnetically confined
  plasmas
• Size and cost of a fusion reactor determined by
  particle and energy confinement time and fusion
  self-heating.
• Plasma turbulence is a complex nonlinear
  phenomenon
• Large time and spatial scale separations similar
  to fluid turbulence.
• Self-consistent electromagnetic fields many-body
  problem
• Strong nonlinear wave-particle interactions
  kinetic effects.
• Importance of plasma spatial inhomogeneities,
  coupled with complex confining magnetic fields,
  as drivers for microinstabilities and the
  ensuing plasma turbulence.

15
The Fundamental Equations for Plasma Physics
BoltzmannMaxwell
6Dtime

• Complete but impractical
• Cannot solve on all time and length scales

• Can eliminate dimensions by integrating over
  velocity space (assuming a Maxwellian)

16
Gyrokinetic Approximation forLow Frequency Modes

• Gyrokinetic ordering
• Gyro-motion guiding center drifts charged ring
• Parallel to B mirror force, magnetically trapped
• Perpendicular E x B, polarization, gradient, and
  curvature drifts
• Gyrophase-averaged 5D gyrokinetic equation
• Suppress plasma oscillation and gyro-motion
• Larger time step and grid size, smaller number of
  particles

17
The Gyrokinetic Toroidal CodeGTC

• Description
• Particle-in-cell code (PIC)
• Developed by Zhihong Lin (now at UC Irvine)
• Non-linear gyrokinetic simulation of
  microturbulence Lee, 1983
• Particle-electric field interaction treated
  self-consistently
• Uses magnetic field line following coordinates
  (y,q,z)
• Guiding center Hamiltonian White and Chance,
  1984
• Non-spectral Poisson solver Lin and Lee, 1995
• Low numerical noise algorithm (df method)
• Full torus (global) simulation

18
The Particle-in-cell Method

• Particles sample distribution function
• Interactions via the grid, on which the potential
  is calculated (from deposited charges).

• The PIC Steps
• SCATTER, or deposit, charges on the grid
  (nearest neighbors)
• Solve Poisson equation
• GATHER forces on each particle from potential
• Move particles (PUSH)
• Repeat

19
Charge Deposition4-point average method
20
Quasi-2D Structure of Electrostatic Potential
21
Global Field-aligned Mesh (Y,a,z) ? a q - z/q
Y
z
z

• Saves a factor of about 100 in CPU time

22
Domain Decomposition

• Domain decomposition
• each MPI process holds a toroidal section
• each particle is assigned to a processor
  according to its position
• Initial memory allocation is done locally on each
  processor to maximize efficiency
• Communication between domains is done with MPI
  calls (runs on most parallel computers)

23
2nd Level of ParallelismLoop-level with OpenMP
24
New MPI-based particle decomposition

• Each domain in the 1D (and soon 2D) domain
  decomposition can have more than 1 processor
  associated with it.
• Each processor holds a fraction of the total
  number of particles in that domain.
• Scales well when using a large number of
  particles

25
Main Computing PlatformNERSCs IBM SP Seaborg

• 416 x 16-processor SMP nodes (with 64G, 32G, or
  16G memory)
• 380 compute nodes (6,080 processors)
• 375 MHz POWER 3 processors with 1.5
  GFlops/sec/proc peak

26
CRAY X1 at ORNL

• 512 Multi-streaming vector processors (MSPs)
• 12.8 Gflops/sec peak performance per MSP
• Currently being upgraded to X1E (1,024 18GF/MSP)

27
Earth Simulator

• 5,120 vector processors
• 8 Gflops/sec per proc.
• 40 Tflops/sec peak

• Collaboration with Dr. Leonid Oliker of LBL/NERSC
• Only US team doing performance study on ES
• Many thanks to Dr. Sato

28
Optimization Challenges

• Gather-Scatter operation in PIC codes
• The particles are randomly distributed in the
  simulation volume (grid).
• Particle charge deposition on the grid leads to
  indirect addressing in memory
• Not cache friendly.
• Need to be tuned differently depending on the
  architecture.

particle array scatter operation
grid array
29
Vectorization Work

• Main challenge charge deposition (scatter)
• Need to avoid memory dependencies
• Solved with work-vector method
• Each element in the processor register has a
  private copy of the local grid
• ES Minimize memory banks conflicts
• Use duplicate directive (thanks to David
  Parks)
• X1 Streaming vector
• Straightforward since GTC already had loop-level
  parallelism.

30
GTC Performance
of proc. part (Bil-lion) IBM SP 3 (Seaborg) IBM SP 3 (Seaborg) Itanium 2 Quadrics Itanium 2 Quadrics CRAY X1 CRAY X1 ES ES
of proc. part (Bil-lion) Gflop Pk Gflop Pk Gflop Pk Gflop Pk
64 0.207 9.0 9.3 25.0 6.9 82.6 10.1 102.4 20.0
128 0.414 17.9 9.3 49.9 6.9 156.2 9.6 199.7 19.5
256 0.828 35.8 9.3 97.3 6.9 299.5 9.1 396.8 19.4
512 1.657 71.7 9.4 194.6 6.8 783.4 19.1
1024 3.314 143.4 8.7 378.9 6.7 1,925 23.5
2048 6.627 266.2 8.4 757.8 6.7 3,727 22.7
3.7 Teraflops achieved on the Earth Simulator
with 2,048 processors using 6.6 billion
particles!!
31
Performance Results
32
(No Transcript)
33
Device-size Scans ITER-size Simulations

• ITER-size simulation using 1 billion particles
  (GC), 125 M spatial grid points, and 7000 time
  steps --- leading to important (previously
  inaccessible) new results
• Made possible by mixed-model MPI-OpenMP on
  Seaborg

34
Continuous Improvements in GTC bringnew
Computational Challenges

• Recent full kinetic electron simulations of
  electron temperature gradient instability
  required 8 billion particles!
• Electron-wave interaction has sharp resonances
  that requires higher phase space resolution
• Fully electromagnetic version requires new solver
  (multi-grid)

35
Look for many GPS-related work during this
conference

• Scientific accomplishments with enhanced versions
  of GTC (Z. Lin et al., presented by G. Rewoldt)
• Shaped plasma device simulations with general
  geometry GTC (W. Wang)
• New electromagnetic solver for kinetic electrons
  capability (M. Adams)
• Visualization techniques (K.-L. Ma)
• Data management and workflows (S. Klasky)

36
Conclusions

• Simulating fusion experiments is very challenging
• It involves multiscale physics
• Gyrokinetic particle-in-cell simulation is a very
  powerful method to study plasma micro-turbulence
• The GTC code can efficiently use the available
  computing power
• New and exciting discoveries are continuously
  being made with GTC through advanced computing