30 Years of Navy Modeling and Supercomputers: an Anecdotal History

1
30 Years of Navy Modeling and Supercomputers an
Anecdotal History
Tom Rosmond Marine Meteorology Division Naval
Research Laboratory Monterey, California USA
2
Outline

• Before my Time 1961-1974
• My Early Years 1974-1980
• The Golden Years of NWP late 1970s-early 1980s
• Up to the Present
• Conclusions and Thoughts for the Future

3
Before my Time 1961-1974
THE BEGINNING

• 1959 Capt Wolff sets up shop on NPS Campus,
  Monterey
• CDC 1604 Seymour Crays first design with
  CDC (first supercomputer?)
• 1961 Establishment of Fleet Numerical Weather
  Facility (FNWF)
• 1964 Routine dissemination of numerical products
• Successive correction analyses of SLP, upper
  air heights
• NH Barotropic and thickness advection models

4
Before my Time 1961-1974
CDC 6500s and NHPE

• 1967 FNWC procurement of CDC 6500 to replace
  1604
• Dual-processors, comparable to 386 microprocessor
  in FP performance
• Supercomputer of time was Seymour Crays CDC 6600
  (NMC in 1966)
• 1969 Second 6500 acquired
• 1970 Northern Hemisphere Primitive Equation
  (NHPE) model operational on 4 processors.
• Worlds first multi-processor production code

5
Before my Time 1961-1974
4-processor 6500 NHPE

• Developed by P. Kessel and F. Winninghoff
• Processors shared model data through extended
  core storage (ECS)
• OS modifications to allow processor
  synchronization
• Similarities to both shared memory (OpenMP) and
  distributed memory (MPI) programming models
• Parallel efficiency 75
• An impressive achievement that is certainly
  underappreciated
• An example of the struggle it was to fit models
  onto these early systems

6
My Early Years 1974-1980

• 1974 I joined the Environmental Prediction
  Research Facility (EPRF)1
• Little involvement in FNWC2 model development
• Access of FNWC computer systems (6500s, 1604)
• 1976 NEPRF Numerical Modeling Department formed
  at request of FNWC CO Capt. R. Hughes
• He realized that FNWC was unable to maintain R
  D continuity needed for NWP system development.
• Need for a global NWP capability was major
  motivation, and he was committed to getting
  computer system to support it.
• 1976 FNWC acquired CDC CYBER 175 (similar to CDC
  7600)

1My organization EPRF NEPRF NOARL
NRL-Monterey
2Fleet Numerical FNWF FNWC FNOC
FNMOC
7
My Early Years 1974-1980
NEPRF Global NWP System Development

• 1976 Navy Operational Global Atmospheric
  Prediction System (NOGAPS)
• UCLA General Circulation Model
• Barnes Successive Corrections Global Analysis
• Variational Initialization with Balance Equation
  Constraint
• 1977 We also had spectral model dynamical core
• UCLA GCM physics
• Sat on shelf for several years in favor of UCLA
  GCM based system
• 1980 Prototype NOGAPS running on CYBER 175

8
My Early Years 1974-1980

• 1977 Benchmarking for FNWC procurement
• NCAR Cray-1, first system outside Los Alamos?
• UCLA GCM 2.40 x 3.00, 6 levels
• First introduction to vector programming
• Crude compiler non-vectorizable code poor
  performance
• Overly conservative target performance, CYBER
  203 could compete
• Allowed subsequent CDC success with CYBER 205
• 1980 CYBER 203 delivered to FNOC
• Heroic work by CDC to get UCLA model to run fast
• 1982 CYBER 203 replaced with CYBER 205

9
The Golden Years of NWP1st half late 70s mid
80s

• Supercomputers were cheapest computers you could
  buy (price/performance)
• Cray Cray-1, XMP, etc
• CDC CYBER 203/205, ETA-10
• IBM 360-195VP
• Development of global NWP forecast systems
• Dominance of spectral models
• Increased realization of importance of data
  assimilation
• Establishment of ECMWF
• Raised the stakes in operational NWP
• Accelerated progress
• Provided gold standard

10
The Golden Years of NWPNavy operational models
NOGAPS UCLA-GCM - 2.40x3.00 x L9 1-pipe/8Mbyte
C205 25min/fcstday Spectral T47L18
2-pipe/32Mbyte C205 6min/fcstday Spectral
T79L18 2-pipe/32Mbyte C205 25min/fcstday
(32bit) Spectral T79L24 4-pipe/64Mbyte C205
12min/fcstday (32bit) NORAPS (developed by Rich
Hodur, NEPRF) Globally relocatable regional
model 6-8 areas run operationally Varying
resolution, domain and grid sizes e.g. ran over
South Atlantic during Falklands war (1980)
11
Some special comments about the CYBER 205

• CDC aggressively pursued meteorology market, both
  operations and research
• FNOC, NMC, UKMO, GSFC, GFDL, etc
• CRAY systems were better general purpose, but 205
  excelled on our applications
• Initially 205 was difficult to use, compiler/user
  software was rudimentary, but
• Language extension showed how machine worked
• Rich array of exotic vector hardware instructions
• System software matured
• 32 bit/64 bit floating point support
• Easy mixing of Fortran and explicit vector
  instructions, we could get close to the
  hardware
• Spectacular percentage of peak performance
  possible

12
The Golden Years of NWP2nd half mid 80s
early 90s

• Introduction of multi-processor systems
• Divide and conquer
• X/MP, Y/MP, ETA-10
• Multi-tasking/vectorization programming model
  (parallel/vector)
• Spelled the end of single processor
  supercomputers, e.g. CYBER 205
• Important changes in technology
• Price/performance advantage of supercomputers
  ending
• Introduction of desktop workstations
• Many people didnt need supercomputers, just
  cheap computing
• Shrinking supercomputer market

13
Up to the Presentearly 1990s

• 1990 Introduction of CRAY C90
• The best supercomputer ever?
• Run by a higher percentage of NWP
  operational/research centers than any system
  before or since.
• Parallel/vector programming model very user
  friendly
• Easy to get high percentage of peak performance
• 1991 8 processor C90 at FNMOC
• NOGAPS T159L24 6 processors 10 min/fcstday
• 1996 8 and16 processor C90s at FNMOC
• NOGAPS T159L24 12 processors 6 min/fcstday
• 1997 COAMPSTM replaced NORAPS as Navys
  regional forecast system

14
Up to the PresentMid to Late 1990s

• Beginnings of dramatic changes in supercomputer
  industry
• Scalable commodity based architectures appearing,
  e.g. CRAY T3E
• Powerful workstations replacing supercomputers
  for many applications
• End of domination by American supercomputer
  vendors
• CDC/ETA were long gone
• CRAY sold to SGI
• MTTB (mean time to bankruptcy) very short for new
  companies
• Rise of Japanese vendors to dominate NWP
  marketplace, at least outside U.S.
• Fujitsu
• NEC

15
Up to the Present2000 to today

• Proliferation of commodity based, scalable
  architectures for NWP applications
• Many T3Es still in use
• SGI Origin 2000, Origin 3000
• IBM SP3, SP4
• Linux clusters are also viable alternatives,
  especially when price/performance is overriding
  issue
• But, rumors of the demise of vector architectures
  are exaggerated
• 2000 NEC only vendor with traditional vector
  architecture, SX-6, but
• 2002 Resurrected CRAY, Inc introduced X-1
• Price/performance vs ultimate performance is
  central question concerning long-term prospects
  of these systems

16
Up to the Present2000 to today

• 2001 FNMOC replaced C90s with O3000s
• 1152 total processors
• NOGAPS T239L30 120 processors 6 min/fcstday
• COAMPSTM 8-11 areas 40-180 processors 10-30
  min/fcstday
• 2003 Operational NAVDAS/NOGAPS (3DVAR) 60
  processors
• 2004 Direct radiance retrievals with NAVDAS
• 2004 Currently under development at NRL,
  Monterey
• T479L54 semi-Lagrangian NOGAPS
• NAVDAS-AR 4DVAR extension of NAVDAS
• COAMPSTM/WRF more areas, higher resolution
• Clearly computational requirements are never
  satisfied!!!

17
Conclusions and thoughts for the future

• Vector architectures will continue to be viable
  candidates for NWP applications
• Sharing some features with commodity systems,
  e.g. caches
• When ultimate performance overrides
  price/performance
• Will be part of heterogeneous computing
  environments
• Scientific computing ( and therefore NWP) is now
  niche market
• Be thankful that video game applications share
  many of our requirements
• Consumer based industry has driven some hardware
  costs to astonishingly low levels, e.g. hard
  drives
• We must work with vendors to ensure that our
  requirements are not forgotten
• Fortran compilers
• High-performance interconnects