PowerPoint Presentation Acceleration of Cosmic Rays at Large Scale Shocks and their Implication for

1
Shedding Light on a Dark Universe with
Supercomputing
Francesco Miniati (ETH-Z) April 28 2008, USI
Lugano
2
Outline

• Big Bang Model
• Adaptive Mesh Refinement for Cosmology/Astrophysi
  cs
• Physics Algorithms (stiff sources, cosmic-rays)
• Conclusions

3
What is Physical Cosmology ?
Cosmology is the scientific study of the large
scale properties of the Universe as a whole to
understand, in particular, the origin, evolution
and ultimate fate of the entire Universe.
And how does it proceed ?

• We cant do experiments on the Universe, e.g.
  cant change the initial conditions and see what
  happens.We can only observe what is the Universe
  is like.
• But because light travels at a finite speed, we
  see objects far away at an earlier time. We can
  use telescopes as time machines.
• So we study what past, present and future
  conditions of the Universe are compatible with
  our observations and the same laws of physics
  that apply in our laboratories.

4
Big-Bang Cosmological Model
Gravity is the only long range force without
self-shielding, so it dominates the large scale
dynamics in the Universe.

• Big-Bang model rests on two theoretical pillars
• Theory of General Relativity Matter tells space
  how to curve and space tells matter how to move
  (J. Wheeler)
• Cosmological Principle on large enough scales
  the universe is homogeneous and isotropic

5
Space-Time Geometry

• Total Energy lt 0
• Sum of angles gt 180
• Positive curvature
• VFinite
• Total Energy gt 0
• Sum of angles lt 180
• Negative curvature
• VInfinite
• Total Energy 0
• Sum of angles 180
• No curvature
• VInfinite

6
Einsteins Biggest Blunder

• Einstein created a general relativistic model for
  the Universe, based on what was known in 1917
  ALMOST NOTHING !
• One fact was known in 1917 the sky is dark at
  night. And Einstein ignored it.
• General Relativity predicted an expanding or
  contracting Universe, which was thought absurd
  back in 1917.
• The Universe was thought to be static, so
  Einstein added a new constant to his equations
  for gravity the cosmological constant, ?.

7
Expansion of the Universe
8
Discovery of Dark Matter
In 1933 Swiss astronomer Fritz Zwicky pointed
out the existence of missing matter but was
ignored for nearly 40 years!
Coma cluster of galaxies
When researchers talk about neutron stars, dark
matter, and gravitational lenses, they all start
the same way Zwicky noticed this problem in
the 1930s. Back then, nobody listened . . .
(Maurer, S.M., 2001)
9
Data on velocity vs distance is now much better!
1995
2004
10
Cosmic Pie
11
The Large Scale Structure of the Universe
2dFGalaxy Redshift Survey 2003
12
Cosmic Microwave Background
Penzias Wilson (1965)
COBE (1992)
WMAP (2005)
13
Galaxy Formation/Evolution
Astrophysical problems to be addressed with same
technology
Formation of Planets
Origin of Cosmic Magnetism
Star Formation
Turbulence
14
Governing Equations
eqs. flops/cell
Hydrodynamics (baryonic matter)
PPM gt6 6000
Radiative cooling (galaxy formation), Magnetic
Fields, Cosmic-rays, Viscosity
Flow of Collisionless Dark Matter Particles in
position and velocity space
Gravity (structure formation)
Multigrid 1 1700
15
Challenges
-large dynamic range -large memory
requirements -scalability -rich physics (e.g.
magnetic fields, cosmic rays, radiative
transfer) -stiffness (energy losses, drag,
radiation pressure) -visualization, real-time
visualization, monitoring and steering (Ernst
Mach "seeing is understanding)
16
Formation of a galaxy in cosmological context

• Memory requirements
• Lbox gt 1.5?108 ly tidal effects
• ?x lt 30 ly to resolve
  proto-stellar clouds
• dynamic range of spatial scales gt 5?106
• fixed gridsgt1020 vol. elements, i.e. need los of
  memory !

Time steps requirements
N
Steps Solar System lt
10 1013 Planet Formation
107-8
108-9 Cosmology
109-10 104-5
Entering petaflop regime
17
We need to compare the observations with a
realistic computer model of ICM turbulence.What
does it take ?
18
Fundamental differences between SPH and grid
methods Agertz, Moore, Stadel, Potter, Miniati,
et al. (MNRAS 2007)
grid methods are the best way to follow
gravitational clustering until it happens J.
Barnes SPH is the best way to follow
hydrodynamics until it happens G. Lake
19
Adaptive Mesh Refinement Cosmological Code
(Miniati Colella 2007, JCP )

• Hybrid gas particles
• - C abstraction, encapsulation, polymorphism,
  modularity
• FORTRAN77 high performance bulk float. point
  intensive ops.
• - MPI parallel low level style for high
  communication perform.
• HDF5 parallel I/O, platform independent
• - Based on CHOMBO-AMR infrastructure (P. Colella,
  LBL/NERSC)
• Hydrodynamics
• - PPM dimensionally unsplit (Colella 1990)
  explicit, BCs once/step
• - uses entropy equation for handling hypersonic
  flows
• Particle pusher
• - time centered, modified symplectic scheme (KDK,
  DKD)

20
Gravity - multilevel-multigrid solver long
range, global communication Sources - scheme
for stiff sources (Miniati Colella 2007, JCP) -
ionizing background, radiation cooling, star
formation Cosmic Rays CR acceleration at shocks
? losses transport (Miniati 2001, CPC 2007,
JCP) MHD In progress
21
Adaptive Mesh Refinement (AMR, Berger Oliver
1984)

• locally refine patches where need to improve the
  solution
• each patch is a rectangular structured grid
• better data access
• reduced overhead of additional AMR operation
• grid patches are dynamically created and
  destroyed

22
(No Transcript)
23
Refinement in Time as well as in Space
(sub-cycling)

• Advantages
• better efficiency (CFL condition)
• Everything at same CFL number improved
  advection performance
• Disadvantage
• Takes extra work!

(Berger Oliver 1984)
24
Time sub-cycling
AMR N-updates Dom. Vol.
w/o subcycl. Level 0 1.5 x 107
643 1 9.6 x 108
1 3.1 x 107 1283 1/8
1.0 x 109 2 1.6 x 108
2563 1/24 2.5 x 109
3 3.5 x 108 5123 1/180
2.8 x 109 4 4.7 x 108
10243 1/2000 1.9 x 109 5
4.3 x 108 20483 1/36000
8.6 x 108 6 2.4 x 108 40963
1/1000000 2.4 x 108 GT 1.7 x
109
1 x 1010
25

• AMR Design Issues
• Coupling coarse and fine level solutions
• Fine grids need boundary conditions form coarse
  grids
• interpolated in time and space
• coarse grids need to see the effects of fine
  grids
• Synchronization of coarse and fine level
  solutions
• maintain constraints in the presence of AMR
• Flux conservation
• Divergence constraint (MHD)
• Freestream (incompressible flows)
• Discontinuous grid spacing at coarse/fine level
  boundary
• Failure of error cancellation can lead to large
  truncation error

26
Divergence theorem is applied in space-time to
obtain discrete evolution equation
?
27
AMR for Hyperbolic Conservation Laws (Berger
Colella 1989)

• Conservative averaging coarse average fine
  solution onto coarse grid
• Refluxing correct coarse cells adjacent to fine
  grid to maintain flux conservation

28

Discretized Elliptic PDEs
Naïve approach Solve ??c?c on coarse grid
Solve ??f?f on fine grid using coarse grid
values to interpolate boundary conditions. Solutio
n converges as hc, i.e. no benefit from local
refinement !
29
Properly posed problem Solve ?c?comp?c on
coarse grid - C(fine grid) Solve ?f?comp?f on
fine grid ?0 ??/?n 0 on coarse/fine
interface Use quadratic (3rd order) interpolant
for fine grid boundary conditions
30
Particles

• Transfer particles across coarse/fine
  boundaries need to respect subcycling.
• particle mesh and mesh particle at coarse/fine
  boundaries buffer zones.
• particle images on non valid grids (covered by
  finer grids) aggregation.

31
Weak Scaling Benchmarks
Colella et al. 2007

• Asynchronous local nonlinear effects can lead to
  load imbalance
• I/O

32
Stiff Sources
(FM Colella, 2007, J. Comp. Phys., 224, 519)
33
Primitive formulation
34
Operator splitting delivers only first order
accuracy.
(Strang 1968, LeVeque 1997)
35
Duhamels formula
36
(No Transcript)
37
Convergence rates Transverse wave
Mach 10 shock-cloud interaction
38
CR Hydrodynamics

• Why ? Because CR are dynamically important in
  many astrophysical plasmas need numerical
  modeling
• Add another dimension to the problem, namely
  momentum space
• Their effect ? They change the hyperbolic
  structure of the equations modify shock
  structure, jump conditions, speed of sound etc

39
Cosmic-ray mediated shocks
(Mewaldt et al 2001)
Solar Wind
Shock Precursor
Viscous Subshock
40
Diffusion-convection equation

• Divide momentum space in a few coarse bins
• Evolve integral of f(p) over those bins
• Use a sub-bin power-law model for f(p) in each
  bin.

41
Coarse grained DCE
Update in smooth flows
(FM 2001, CPC 141, 17)
42
Time averaged fluxes in momentum space
(FM 2001, CPC 141, 17)
Furthermore we can cast the equations describing
the coupled system of gas and CR fluid in
conservative form, carry out the characteristic
analysis and define a modified Riemann solver to
fully account for their dynamical effects. (FM
2007, JCP 227, 776)
43
The cosmic gamma-ray background
Physics Research Highlight Nature, 2007, 3, 677
IC
p0
LDDE
PLE
Miniati, Koushiappas, Di Matteo ApJL, 667, L1,
2007
44
Generic Future Directions

• Driving purpose is to perform HPC for scientific
  discovery
• Good scaling new physics algorithms, explore
  with unprecedented resolution and better physical
  description models of large scale structure of
  the universe. Support large experimental
  initiatives to probe the nature of Dark Energy
  (national and EU).
• Also address new problems origin of cosmic
  magnetism, role of CRs and magnetic fields in
  astrophysical systems (e.g. galaxy
  formation/evolution)
• Apply to other area, e.g. star formation, planet
  formation
• Continue effort to develop accurate numerical
  algorithms to model rich physics of astrophysical
  systems
• Code development/maintenance to exploit latest
  technologies, parallelization strategies to
  achieve petaflop capability (with Colella _at_
  LBL/NERSC)

45
Final Remarks
Research in Astronomy is currently blooming past
ten years dark energy on the largest scales and
first extra-solar planets discovered on the
smallest scales (M. Mayor, Geneva Obs.) New
discovery --gt new theoretical/computational
challenges for modeling and understanding their
significance. In both above cases technology
drives progress better telescopes and
experimental ingenuity vs better
computing machines and codes/algorithms. Astrophy
sical systems are complex and highly
nonlinear, making numerical simulations crucial
for their modeling. They are characterized by
large a dynamic range and the concurrence of a
variety of physical processes. AMR technique is a
powerful method to attack these problems, equally
well suited for planet formation as well as
cosmological systems.