CCS machine development plan for post-peta scale computing and Japanese the next generation supercomputer project

1
CCS machine development plan for post-peta scale
computingand Japanese the next generation
supercomputer project

• Mitsuhisa Sato
• CCS, University of Tsukuba

2
Computing resources in CCS
PACS-CS
FIRST
(2006)
(2007)
GRAPE-6

• A Special-purpose system to Astrophysics
  simulation by hybrid computation of radiation and
  N-body.
• Each node is equipped by GRAPE-6, which is an
  accelerator specialized for N-body Gravity
  calculation.
• 256 nodes
• performance cluster 3.5TFLOPSGrape-6 35TFLOPS

• node 2560 node
• (Intel Xeon 2.8GHz, single core /node)
• peak performance 14.34 TF
• memory 5 TB
• network 250MB/s/link x 3 (3D-HXB by GbE)

T2K-tsukuba
(2008)

• Designed by T2K Open Supercomputer
• Alliance (U. Tokyo and Kyoto U)
• Spec
• 648 nodes
• (quad Opteron, 4sockets/node)
• 10000 cores
• Peak performance 95.4TF
• total memory 20TB
• total disk capacity 800TB
• ( 20th in top 500, June, 2008)

3
System installation and future plans
H16
H17
H18
H19
H20
H21
H22
H23
H24
2013
CP-PACS
(??)
FCS-IV
HA-PACS (planned)
PACS-CS
FCS-V
FCS Front-end system
2011-2013
T2K
VPP
suspended
the next system to T2K
FIRST
NGS (10PF)
4
Issues for Post-peta scale systems (not exa?)

• System to enable strong-scaling
• the current petascale
• system enabled by weak-scaling
• We need more powerful
• node network
• GPGPU is one of solution
• More specialized architecture
• we need a sharp science target
• All applications cannot use
• More difficult to program
• Need supports from CS-side
• Collaboration with computer science and
• computational science

CCS's mission
5
HA-PACS Highly Accelerated Parallel Advanced
system for Computational Sciences (planned)
Objective to investigate acceleration
technologies for post-petascale computing and its
software, algorithms and computational science
applications, and demonstrate by building a
prototype system

• Design and deploy a GPGPU-based Cluster system
• Research on programming model and languages,
  environment for parallel system with
  accelerators.
• Design of Algorithms and applications for
  parallel system with accelerators.
• Research on architectures for parallel system
  with accelerators.

examples
6
HA-PACS/NG powered by PEARL Link

• PEARL PCI-Express Adaptive and Reliable Link
• Use PCI-Express as a high-speed link
• Connect CPU and devices including
• GPGPU through a router chip,
• PEACH (PCI-Express Adaptive
• Communication Hub)

Infiniband QDR
.....
.........
IB switch
IB switch
Infiniband QDR
..............
PEARL Link
..............
CPU
CPU
GPGPU
GPGPU
GPGPU
12 node
PCIe
PCIe
PEARL Link
GPGPU
GPGPU
7
Strategic target computational sciences of HA-PACS

• ? Bio-physics high performance QM/MM hybrid
  simulation for mechanisms of high-efficiency
  enzymatic reactions, electronic and 3D structures
  of biomacromolecules
• Speedup of QM is a key for this simulations

• ? astrophysics full Hydrodynamics and
  radiative-transfer simulation for the Universe
  and Formation of Astronomical objects
• Full 6 dimensional simulation is required

? Particle physics full-lattice QCD simulation
8
Japanese the next generation supercomputer project
9
background Japanese government plan

• The 3rd Science and Technology Basic Plan
  (FY2006-FY2010)
• Next-generation super computing technology is
  selected as one of key technologies of national
  importance
• Development and installation of the advanced
  high performance supercomputer system
  (10petaflops) ? the Next-Generation Supercomputer
• Development application software
• Establishment of Advanced Computational Science
  and Technology Center (tentative name)
• The 4th Science and Technology Basic Plan
  (FY2011-FY2015) (Now under discussion)
• Exaflops class HPC Technology
• New chip device, software, hardware
• After the election of the House of
  Representatives in the last summer,.
• In the November of the last year, the new
  government party have decided to freeze the plan
  of the development at the screening of government
  projects!!!
• In January of this year, the cabinet have made a
  decision to resume the super computer project.

10
The System Overview of NGS
?Massively Parallel/Distributed Memory
Supercomputer?

• Ultra high-speed/ high-reliable CPU
• Advanced 45nm process technology
• 8cores/CPU, 128GFLOPS
• Error recovery ( ECC, Instruction retry, etc.)
• High performance/highly reliable network
• Direct interconnection network by
  multi-dimensional mesh/torus network
• Expandability and reliablity
• System Software
• Linux OS
• Fortran, C, and MPI libraries
• Distributed parallel file system

Logical 3-dimensional torus network
Courtesy of FUJITSU
11
Configuration of Compute Nodes

• Number of nodes gt 80k
• Number of CPUs gt 80k
• Number of cores gt 640k
• Peak Performance gt 10PFLOPS
• Total Memory Capacity gt 1PB ( 16GB/node )

• Multi-dimensional mesh/torus network
• Peak bandwidth 5GB/s x 2 for each direction of
  logical 3-dimensional torus network
• Peak bi-sectional bandwidth gt 30TB/s

5GB/s x 2
5GB/s x 2
???
CPU 128GFLOPS (8 Core)
Core SIMD(4FMA) 16GFlops
Core SIMD(4FMA) 16GFlops
5GB/s x 2
Core SIMD(4FMA) 16GFlops
Core SIMD(4FMA) 16GFlops
5GB/s x 2
Core SIMD(4FMA) 16GFlops
Core SIMD(4FMA) 16GFlops
Core SIMD(4FMA) 16GFlops
Core SIMD(4FMA) 16GFLOPS
L2 5MB
5GB/s x 2
64GB/s
z
MEM 16GB
y
5GB/s x 2
Logical 3-dimensional torus network for
programming
x
12
The Next-Generation Supercomputer Project
?Schedule
FY2008
FY2009
FY2010
FY2011
FY2007
FY2006
FY2012
Tuning and improvement
Production, installation, and adjustment
Detailed design
System
open to users
Next-Generation Integrated Nanoscience Simulation
Verification
Development, production, and evaluation
Applications
Next-Generation Integrated Life Simulation
Verification
Development, production, and evaluation
Computer building
Construction
Design
Buildings
Research building
Construction
Design
13
The categories of users of NGS

• 1. Strategic Use
• MEXT selected 5 strategic fields from national
  viewpoint.
• Field 1 Life science/Drug manufacture
• Field 2 New material/energy creation
• Field 3 Global change prediction for disaster
  prevention/mitigation
• Field 4 Mono-zukuri (Manufacturing technology)
• Field 5 The origin of matters and the universe
• 2. General Use
• The use for the needs of the researchers
  in many science and technology fields including
  industrial use and educational use

14
Organization for NGS

• Advanced Computational Science and Technology
  Center (ACSTC) (tentative name) will be
  organized at NGS.
• MEXT selects 5 core organizations that lead
  research activities in 5 strategic fields
• ACSTC ? Core research center
• Conducts advanced and basic RD in computational
  science
• Leads cooperation among strategic fields
• Provides key knowledge to 5 organizations in
  strategic fields and another research
  organizations
• 5 core organizations ? Research center in each
  field
• Conducts advanced RD in each field
• CCS was selected as a core organization for
  "Field 5 The origin of matters and the universe"
• particle physics, Astrophysics, nuclear physics
• Collaboration with KEK and National Observatory