Supercomputing Systems at NCAR SC2005

1
Supercomputing Systems at NCARSC2005

• Marc Genty
• Supercomputing Services Group

November 15 - 17, 2005
2
NCAR Mission

• The National Center for Atmospheric Research
  (NCAR) is a federally funded research and
  development center. Together with our partners at
  universities and research centers, we are
  dedicated to exploring and understanding our
  atmosphere and its interactions with the Sun, the
  oceans, the biosphere, and human society. NCAR
  consists of the following
• Computational and Information Systems Laboratory
  (CISL)
• SCD Scientific Computing Division
• IMAGe Institute for Mathematics Applied to
  Geosciences
• Earth and Sun Systems Laboratory (ESSL)
• ACD Atmospheric Chemistry Division
• CGD Climate Global Dynamics Division
• HAO High Altitude Observatory
• MMM Mesoscale Microscale Meterology Division
• TIMES The Institute for Multidisciplinary
  Earth Studies
• Earth Observing Laboratory (EOL)
• HIAPER High-Performance Instrumented Airborne
  Platform
• for Environmental Research
• Research Applications Laboratory (RAL)
• RAP Research Applications Programs
• Societal-Environmental Research and Education
  Laboratory (SERE)
• ASP Advanced Study Program
• CCB Center for Capacity Building
• ISSE Institute for the Study of Society and
  Environment (formerly ESIG)

3
NCAR Science
Turbulence
The Sun
Space Weather
Atmospheric Chemistry
Climate
Weather
More than just the atmosphere from the earths
oceans to the solar interior
4
2005 Climate Simulation Lab Science

• Community Climate System Model (CCSM)
• Modeling Climate Change and Climate Variability
  in Coupled Climate-Land Vegetation Models
  Present, Past, and Future Climates
• 50-year Regional Downscaling of NCEP/NCAR
  Reanalysis Over California Using the Regional
  Spectral Model
• Climate Variability in the Atlantic Basin
• Aerosol Effects on the Hydrological Cycle
• Pacific Decadal Variability due to
  Tropical-Extratropical Interaction
• Predictability of the Coupled Ocean-Atmosphere-Lan
  d Climate System Seasonal-to-Interannual Time
  Scales
• The Whole Atmosphere Community Climate Model
• Decadal to Century Coupled Ocean/Ice Simulations
  at High Resolution (0.2?) Using an Innovative
  Geodesic Grid
• Ocean State Estimation
• Devlopment and Application of Seasonal Climate
  Predictions
• gt http//www.scd.ucar.edu/csl/cslannual0505.html

5
69 Member Universities

• University of Alabama in Huntsville
• University of Alaska
• University at Albany, State University of New
  York
• University of Arizona
• Arizona State University
• California Institute of Technology
• University of California, Berkeley
• University of California, Davis
• University of California, Irvine
• University of California, Los Angeles
• University of Chicago
• Colorado State University
• University of Colorado at Boulder
• Columbia University
• Cornell University
• University of Denver
• Drexel University
• Florida State University
• Georgia Institute of Technology

University of Illinois at Urbana-Champaign Univers
ity of Iowa Iowa State University The Johns
Hopkins University University of
Maryland Massachusetts Institute of
Technology McGill University University of
Miami University of Michigan - Ann
Arbor University of Minnesota University of
Missouri Naval Postgraduate School University of
Nebraska, Lincoln Nevada System of Higher
Education University of New Hampshire, Durham New
Mexico Institute of Mining and Technology New
York University North Carolina State
University The Ohio State University University
of Oklahoma Old Dominion University Oregon State
University Pennsylvania State University
Princeton University Purdue University University
of Rhode Island Rice University Rutgers
University Saint Louis University Scripps
Institution of Oceanography at Stanford
University UCSD Texas AM University University
of Texas at Austin Texas Tech University Universit
y of Toronto Utah State University University of
Utah University of Virginia University of
Washington Washington State University University
of Wisconsin - Madison University of Wisconsin -
Milwaukee Woods Hole Oceanographic
Institution University of Wyoming Yale
University York University http//www.ucar.edu/gov
ernance/members/institutions.shtml
6
SCD Mission

• The Scientific Computing Division (SCD) is part
  of the National Center for Atmospheric Research
  (NCAR) in Boulder, Colorado. The goal of SCD is
  to enable the best atmospheric research in the
  world by providing and advancing high-performance
  computing technologies. SCD offers computing,
  research datasets, data storage, networking, and
  data analysis tools to advance the scientific
  research agenda of NCAR. NCAR is managed by the
  University Corporation for Atmospheric Research
  (UCAR) and is sponsored by the National Science
  Foundation.

7
The NCAR Mesa Lab
8
(No Transcript)
9
In the beginning (1963)
10
CDC 3600 System Overview

• Circuitry Design Seymour Cray
• Clock Speed 0.7 MHz
• Memory 32 Kbytes
• Peak Performance 1.3 MFLOPs

11
Todays NCAR Supercomputers

• Bluesky IBM POWER4 AIX - Production - General
  Scientific Use
• 125-node (50 frame) p690 cluster, 1600 1.3GHz
  CPUs, SP Switch2, 15TB FC disk
• Configured as 76 8-way (LPAR) nodes and 25 32-way
  (SMP) nodes.
• Bluevista IBM POWER5 AIX - Production - General
  Scientific Use
• 78-node p575 cluster, 624 1.9GHz CPUs, HPS
  Switch, 55TB FC disk
• NCAR codes are typically seeing a speedup of
  2x-3x over bluesky
• Frost IBM Blue Gene/L - Single Rack - Pset Size
  32
• Lightning IBM SuSE Linux - Production - General
  Scientific Use
• 132-node AMD64/Xeon cluster, 264 2.2/3.0GHz CPUs,
  Myrinet Switch, 6TB SATA disk
• Pegasus IBM SuSE Linux - Production - Real-Time
  Weather Forecasting
• 70-node AMD64/Xeon cluster, 140 2.2/3.0GHz CPUs,
  Myrinet Switch, 6TB SATA disk
• Coral Aspen Systems SuSE Linux - Production -
  IMAGe Divisional System
• 24-node Nacona cluster, 44 3.2GHz CPUs,
  InifinBand Switch, 6TB SATA disk
• Test Systems Thunder P4/HPS, Bluedawn P4/SP
  Switch2, Otis P5/HPS

12
Bluesky
13
Bluesky System Overview

• IBM POWER4 Cluster 1600
• AIX 5.1, PSSP, GPFS, LoadLeveler
• 125-node (50 frame) p690 cluster
• Compute Node Breakdown 76 8-way (LPAR) 25
  32-way (SMP)
• 1600 1.3GHz CPUs
• SP Switch2 (Colony)
• 15TB FC disk
• General purpose, computational resource

14
Bluesky 32-Way LPAR Usage
15
Bluesky 8-Way LPAR Usage
16
Bluesky Science Highlights

• CCSM3 The Community Climate System Model

• Fully-coupled, global climate model that provides
  state-of-the-art computer simulations of the
  Earth's past, present, and future climate states
• The CCSM3 IPCC (Intergovernmental Plan on Climate
  Change) integrations now include roughly 11,000
  years of simulated climate (19th - 24th
  centuries)
• The CCSM3 control run archive contains 4,500
  years of simulated climate at three resolutions
• http//www.ccsm.ucar.edu/

17
Bluesky Science Highlights

• ARW Advanced Research WRF (Weather Research
  Forecasting) Model

• Next-Generation Mesoscale Numerical Weather
  Prediction System
• http//www.mmm.ucar.edu/index.php

18
Bluevista
19
Bluevista System Overview

• IBM POWER5 Cluster
• AIX 5.2, CSM, GPFS, LSF
• 78-node p575 cluster
• 624 1.9GHz CPUs
• HPS Switch (Federation)
• 55TB FC disk
• General purpose, computational resource
• NCAR codes are typically seeing a speedup of
  2x-3x over bluesky
• The bluevista cluster is estimated to have the
  same sustained computing capacity as the bluesky
  cluster

20
Bluevista Usage (Not Yet In Full Production)
21
Bluevista Science Highlights

• 2005 Nested Regional Climate Model (NRCM)
• Focus To develop a state-of-the-science nested
  climate model based in WRF and to provide this to
  the community
• http//www.mmm.ucar.edu/facilities/nrcm/nrcm.php
• 2005 Limited friendly-user time also allocated
• 2006 General scientific production system
• Will augment bluesky capacity

22
Frost
23
Frost System Overview

• IBM Blue Gene/L
• Single rack
• One I/O node per thirty-two compute nodes (pset
  size 32)
• Service node
• One IBM p630 server
• Two POWER4 1.2GHz CPUs 4GB memory
• SuSE (SLES9), DB2 FixPak9
• Front-end nodes
• Four IBM OpenPower 720 servers
• Four POWER5 1.65GHz CPUs 8GB memory
• SuSE (SLES9), GPFS, COBALT

24
Blue Gene/L At NCAR
Blue Gene/L is jointly owned and managed
collaboratively by NCAR and the University of
Colorado (Boulder Denver). There are
Principal Investigators (PIs) associated with
each research facility, and each PI has a small
group of scientists running on the system. Blue
Gene/L is a targeted system at this time with
allocations split among the three primary
research facilities.
25
Bluesky / Frost Side-By-Side
Processors 1600 Peak Teraflops 8.3 Linpack 4.2
Power (kW) 400
Processors 2048 Peak Teraflops 5.73 Linpack 4.
6 Power (kW) 25
The average personal computer consumes about
0.12 kW
26
Frost Usage
27
Frost Principal Research Areas

• Climate and Weather Simulation
• http//www.ucar.edu/research/climate/
• http//www.ucar.edu/research/prediction/
• Computational Fluid Dynamics and Turbulence
• http//www.image.ucar.edu/TNT/
• Coupled Atmosphere-Fire Modeling
• http//www.ucar.edu/research/climate/drought.shtml
  
• Scalable Solvers
• http//amath.colorado.edu/faculty/tmanteuf/
• Aerospace Engineering
• http//icme.stanford.edu/faculty/cfarhat.html

28
Frost Science Highlights

• Modeling Aqua Planet on Blue Gene/L
• Dr. Amik St-Cyr - Scientist
• Computational Science Section
• Scientific Computing Division

• NCAR Booth 100PM - Tuesday, November 15, 2005
• NCAR Booth 100PM - Thursday, November 16, 2005

29
Lightning
30
Lightning System Overview

• IBM Cluster 1350
• SuSE (SLES9) Linux, CSM, GPFS, LSF
• 132-node AMD64/Xeon cluster
• 264 2.2/3.0GHz CPUs
• Myrinet Switch
• 6TB SATA disk
• General purpose, computational resource

31
Lightning Usage
32
Lightning Science Highlights

• TCSP Tropical Cloud Systems and Processes field
  research investigation

• Joint NCAR/NASA/NOAA study of the dynamics and
  thermodynamics of precipitating cloud systems,
  including tropical cyclones
• http//box.mmm.ucar.edu/projects/wrf_tcsp/

33
Lightning Science Highlights

• TIME-GCM Thermosphere Ionosphere Mesosphere
  Electrodynamics General Circulation Model

• Distributed memory parallelism using eight nodes
  (16 MPI tasks)
• Run completed in 12 jobs at 5-6 wallclock hours
  each for a total of 70 hours (12 minutes per
  simulated day)
• http//www.hao.ucar.edu/Public/models/models.html

34
Pegasus
35
Pegasus System Overview

• IBM Cluster 1350
• SuSE (SLES9) Linux, CSM, GPFS, LSF
• 70-node AMD64/Xeon cluster
• 140 2.2/3.0GHz CPUs
• Myrinet Switch
• 6TB SATA disk
• Essentially a 0.5 scale model of the lightning
  cluster
• Real-Time Weather Forecasting

36
Pegasus Usage
37
Pegasus Science Highlights

• AMPS Antarctic Mesoscale Prediction System
  (Polar MM5)

• Twice daily operational forecasts for the
  Antarctic Region (McMurdo Station, Antarctica)
• Sponsored by the NSF Office of Polar Programs
• http//box.mmm.ucar.edu/rt/mm5/amps/

38
Coral
39
Coral System Overview

• Aspen Systems 24-node Nacona cluster (48
  3.2/3.6GHz CPUs)
• SuSE (SLES9) Linux, ABC, NFS, LSF
• Two HP Visualization Nodes (RedHat Enterprise
  Linux V3)
• InifiniBand Switch
• 6TB SATA disk
• Dedicated resource belonging to the Institute for
  Mathematics Applied to the Geosciences (IMAGe)
  Division
• http//www.image.ucar.edu/

40
Tempest

• SGI Origin 3800
• IRIX 6.5.25, NQE
• No cluster mgt s/w or parallel file system
• 128 500-MHz R14000 CPUs
• 64GB Distributed Shared Memory
• NUMAlink Interconnect
• 8.5TB Ciprico SATA RAID disk
• General purpose, post-processing and data
  analysis server
• Managed by the Data Analysis Services Group (DASG)

41
Tempest Science Highlights

• Desktop Techniques for the Exploration of
  Terascale-sized Turbulence Data Sets
• John Clyne - Senior Software Engineer
• High-Performance Systems Section

• NCAR Booth 400PM - Tuesday, November 15, 2005
• NCAR Booth 300PM - Thursday, November 17, 2005

42
Blackforest(318 WHII Node RS/6000 SP) R.I.P. -
12Jan05 _at_ 8am
43
Blackforest Highlights

• 5.4 Year Lifetime
• 30.5 Million CPU Hours Of Work
• 600,000 Batch Jobs
• 50 CPU Hours/Job (On Average)
• 27.28 CPUs (7 Nodes) - Average Job

44
NCAR Supercomputer Performance Numbers
45
Conclusion Questions

• Read more about it
• http//www.scd.ucar.edu/main/computers.html
• Questions or Comments?
• Special thanks to
• Lynda Lester / Pam Gillman (SCD) Photographs
• Tom Engel (SCD) Utilization Charts / Stats
• Irfan Elahi / John Clyne (SCD) Fun Facts
• Ben Foster (HAO) TIME-GCM Data
• Sean McCreary (CSS/CU) BG/L Research Areas
• BJ Heller (SCD) Production Assistance