Architecture of the Earth System Modeling Framework

1
Architecture of the Earth System Modeling
Framework
GFDL FMS Suite
MITgcm

NASA GSFC PSAS
NCEP Forecast
V. Balaji / GFDL, Cecelia DeLuca / NCAR, Chris
Hill / MIT, Max Suarez, Arlindo da Silva / NASA
GSFCEGS - AGUNice, France
NSIPP Seasonal Forecast
NCAR/LANL CCSM
2
Outline

• ESMF Project Overview
• Related Projects and Interactions
• ESMF Design Overview
• ESMF Design Principles
• Timeline and Status

3
Technological Trends
In climate research and NWP... increased
emphasis on detailed representation of individual
physical processes requires many teams of
specialists to contribute components to an
overall modeling system. In computing
technology... increase in hardware and software
complexity in high-performance computing, as we
shift toward the use of scalable computing
architectures and open community software
standards.
4
ESMF Project Overview

• GOAL To increase software reuse,
  interoperability, ease of use and performance
  portability in climate, weather, and data
  assimilation applications
• PRODUCTS
• Coupling superstructure and utility
  infrastructure software
• Synthetic code suite for validation and
  demonstration
• Set of 15 ESMF-compliant applications (including
  CCSM, WRF, GFDL models MIT, NCEP and NASA data
  assimilation systems)
• Set of 8 interoperability experiments
• RESOURCES 10.1M over 3 years from NASA Earth
  Science Technology Office

5
ESMF Collaborators
NOAA NCEPStephen LordMark IredellMike
YoungWeiyu YangJohn Derber MITJohn Marshall,
PIChris Hill NASA DAOArlindo da Silva,
PILeonid ZaslavskyWill SawyerCarlos
CruzUniversity of MichiganQuentin Stout

• NOAA GFDLAnts LeetmaaV. BalajiRobert
  HallbergShep Smithline
• NASA NSIPPMax Suarez Michele RieneckerChristian
  KeppenneAtanas Trayanov
• DOE ANL
• Rob Jacob
• Jay Larson

NSF NCARTim Killeen, PIByron BovilleCecelia
DeLucaRoberta JohnsonJohn MichalakesAl
KellieJeff AndersonDavid NeckelsEarl Schwab
Nancy CollinsJon WolfeSilverio VasquezDennis
FlaniganRobbie Staufer DOE LANLPhil Jones
6
ESMF Interoperability Demonstrations
COUPLED CONFIGURATION NEW SCIENCE ENABLED
GFDL B-grid atm / MITgcm ocn Global biogeochemistry (CO2, O2), SI timescales.
GFDL MOM4 / NCEP forecast NCEP seasonal forecasting system.
NSIPP ocean / LANL CICE Sea ice model for extension of SI system to centennial time scales.
NSIPP atm / DAO analysis Assimilated initial state for SI.
DAO analysis / NCEP model Intercomparison of systems for NASA/NOAA joint center for satellite data assimilation.
DAO CAM-fv / NCEP analysis Intercomparison of systems for NASA/NOAA joint center for satellite data assimilation.
NCAR CAM Eul / MITgcm ocn Improved climate predictive capability climate sensitivity to large component interchange, optimized initial conditions.
NCEP WRF / GFDL MOM4 Development of hurricane prediction capability.
7
Outline

• ESMF Project Overview
• Related Projects and Interactions
• ESMF Design Overview
• ESMF Design Principles
• Timeline and Status

8
Related Projects
ESMF Earth System Modeling Framework CCA DOE
Common Component Architecture SciDAC DOE/NSF CCSM
SciDAC Project GEMS Goddard Earth Modeling
System FMS GFDL Flexible Modeling System SWMF
Space Weather Modeling Framework WRF Weather
Research and Forecast Model CCSM Community
Climate System Model PRISM Program for Int.
Earth System Modeling
CCA
SciDAC
PRISM
WRF
CCSM
ESMF
GEMS
FMS
SWMF
9
ESMF and PRISM

• A European Earth system modeling infrastructure
  project, started December 2001
• Funded by the European Commission (4.8M)
• Involves current state-of-the-art atmosphere,
  ocean, sea-ice, atmospheric chemistry,
  land-surface and ocean-biogeochemistry models
• 22 partners leading climate researchers and
  computer vendors, includes MPI, KNMI, UK Met
  Office, CERFACS, ECMWF, DMI.

10
ESMF and PRISM, cont.

• Working together to supplement CF convention for
  physical field names and quantities
• Component interface database developed by ESMF
  stores fields in model import and export states
• V. Balaji (GFDL) is ESMF liaison to PRISM

11
ESMF and CCA

• Common Component Architecture (CCA) is creating a
  minimal interface and sets of tools for linking
  high performance components. CCA can be used to
  implement frameworks and standards developed in
  specific domains (such as ESMF).
• DOE funded through SciDAC.
• Collaborators include LANL, ANL, LLNL, ORNL,
  Sandia, University of Tennessee, and many more.
  Ongoing ESMF collaboration with CCA/LANL on
  language interoperability.
• Shujia Zhou / Arlindo da Silva (NASA GSFC)
  currently prototyping an ESMF API using CCA tool
  CCAFFEINE working towards CAM/DAO PSAS coupling
  using ESMF and CCA.

12
Outline

• ESMF Project Overview
• Related Projects and Interactions
• ESMF Design Overview
• ESMF Design Principles
• Timeline and Status

13
Architecture
Coupling Layer
ESMF Superstructure
Model Layer
User Code
Fields and Grids Layer
ESMF Infrastructure
Low Level Utilities
External Libraries
BLAS, MPI, NetCDF,
14
ESMF Components

1. ESMF provides an environment for assembling
   geophysical components into an application.

Application Component
Gridded Components
Coupler Components

• ESMF provides a toolkit that components use to
• increase interoperability
• improve performance portability
• abstract common services

Component Init( ), Run( ), Checkpoint( )
Field Halo( ), Import( ), Export( ) I/O
Grid Regrid( ), Transpose( ) Metrics
Layout, PE List, Machine Model
15
General Features

• ESMF will be usable by models written in
  F90/C/C
• ESMF will be usable by models requiring adjoint
  capability
• ESMF will be usable by models requiring shared or
  distributed memory parallelism semantics
• ESMF will support SPMD and MPMD coupling
• ESMF will support several I/O formats, including
  GRIB/BUFR, netCDF, HDF
• ESMF will have uniform syntax across platforms
• ESMF will target a broad range of platforms, from
  major centers ? commodity hardware

16
Outline

• ESMF Project Overview
• Related Projects and Interactions
• ESMF Design Overview
• ESMF Design Principles
• Timeline and Status

17
Design PrinciplesScalable Applications
Since each ESMF application is also a component,
entire ESMF applications may be treated as
Gridded Components and nested within larger
applications.
climate_comp
Example atmospheric application containing
multiple coupled components within a larger
climate application
ocn2atm_coupler
ocn_comp
atm_comp
phys2dyn_coupler
atm_phys
atm_dyn
PE
18
Design PrinciplesLocal Communication
All inter-component communication within ESMF is
local.
climate_comp
This meansCoupler Components must be defined
on the union of the PEs of all the components
that they couple. In this example, in order to
send data from the ocean component to the
atmosphere, the Coupler mediates the send.
atm2ocn _coupler
ocn_comp
atm_comp
phys2dyn_coupler
atm_phys
atm_dyn
PE
19
Design PrinciplesModularity
Gridded Components dont have access to the
internals of other Gridded Components. Gridded
Components may - pass their States through their
argument list or - receive methods, through their
argument list, for interacting with other
Components (Transforms).
call ESMF_CompRun(atm, xform) call
ESMF_CompRun(ocn, xform)
coupler
transform
ocn_component call ESMF_StateXform(ex_state,
xform)
atm_component call ESMF_StateXform(xform,
im_state)
20
Design PrinciplesUniform Communication API
The same programming interface is used for shared
memory, distributed memory, and combinations of
both. Machine model abstraction of machine
architecture (num_nodes, num_pes_per_node) DE a
decomposition element - may be virtual, thread or
MPI process Layout an arrangement of DEs, in
which dimensions requiring faster communication
may be specified and resources arranged
accordingly
A 2 x 6 Layout of 4 3-processor nodes
The data in a Grid is decomposed into the number
of chunks specified in the Layout.
21
ESMF Class Structure
GriddedComponent
CouplerComponent
Superstructure
State
Infrastructure
Bundle
Regrid
Field
Grid
PhysGrid
DistGrid
F90
Data
Communications
Route
Layout
Array
Comm
C
More work Prototype exists
MachineModel
Utilities TimeMgr, LogErr, I/O etc.
22
Outline

• ESMF Project Overview
• Related Projects and Interactions
• ESMF Design Overview
• ESMF Design Principles
• Timeline and Status

23
Timeline
May 2002 Draft Developers Guide and Requirements Document completed Community Requirements Meeting and review held in D.C.
July 2002 ESMF VAlidation (EVA) suite assembled
August 2002 Architecture Document major classes and their relationships Implementation Report language strategy and programming model Software Build and Test Plan sequencing and validation
May 15 2003 First API and software release, Community Meeting at GFDL
July 2003 First 3 interoperability experiments completed
April 2004 Second API and software release, Community Meeting
July 2004 All interoperability experiments completed
Jan 2005 Final release
Currently planning for ongoing funding and
extensions
24
May 2003 Release

• Focus for May 2003 ESMF release is on
  developing sufficient infrastructure and
  superstructure to achieve the initial set of
  interoperability experiments.
• These are
• FMS B-grid atmosphere coupled to MITgcm ocean
• CAM atmosphere coupled to NCEP analysis
• NSIPP atmosphere coupled to DAO analysis

25
More information

• 2nd ESMF Community Meeting at GFDL May 15, 2003
• ESMF website http//www.esmf.ucar.edu
• On the website
• Information about the Community Meeting
• ESMF documents
• ESMF browsable source code repository
• On-line forms for community input and catalogued
  responses
• Milestone schedule, project archives and history,
  related projects
• Management plan, board and team contacts
• More