Update on the Earth System Modeling Framework

1
Update on the Earth System Modeling Framework
MITgcm
GFDL FMS Suite

NCAR/LANL CCSM
NCEP Forecast
Cecelia DeLuca, NCARCCSM Software Engineering
Working Group MtgBoulder, CO
NSIPP Seasonal Forecast
2
Outline

• ESMF Project Overview
• Related Projects and Interactions
• ESMF Design Overview
• ESMF Design Principles
• Timeline and Status

3
Technological Trends
In climate research and NWP... increased
emphasis on detailed representation of individual
physical processes requires many teams of
specialists to contribute components to an
overall modeling system. In computing
technology... increase in hardware and software
complexity in high-performance computing, as we
shift toward the use of scalable computing
architectures.
4
ESMF Project Overview

• GOAL To increase software reuse,
  interoperability, ease of use and performance
  portability in climate, weather, and data
  assimilation applications
• PRODUCTS
• Coupling superstructure and utility
  infrastructure software
• Synthetic code suite for validation and
  demonstration
• Set of 15 ESMF-compliant applications (including
  CCSM, WRF, GFDL models MIT, NCEP and NASA data
  assimilation systems)
• Set of 8 interoperability experiments
• RESOURCES 10.1M over 3 years from NASA Earth
  Science Technology Office

5
ESMF Collaborators
NOAA NCEPStephen LordMark IredellMike
YoungWei YuJohn Derber University of
MichiganQuentin Stout MITJohn Marshall,
PIChris Hill DOE Los Alamos National LabPhil
Jones

• NOAA GFDLAnts LeetmaaV. BalajiRobert
  Hallberg
• NASA NSIPPMax Suarez Michele RieneckerChristian
  KeppenneAtanas Trayanov
• DOE ANL
• Rob Jacob
• Jay Larson

NSF NCARTim Killeen, PIByron BovilleCecelia
DeLucaRoberta JohnsonJohn MichalakesAl
KellieJeff AndersonDavid NeckelsNancy
CollinsJon WolfeEarl Schwab NASA DAOArlindo
da Silva, PILeonid ZaslavskyWill SawyerCarlos
Cruz
6
ESMF Interoperability Demonstrations
7
Outline

• ESMF Project Overview
• Related Projects and Interactions
• ESMF Design Overview
• ESMF Design Principles
• Timeline and Status

8
Hierarchy of Modeling Infrastructure Projects
Single Earth system model or modeling system
CCSM Coupled System, NASA GEMS, GFDL FMS
Space Weather (SWMF), Geophysics, Ecosystem
Dynamics,
All Earth, Space, and Life Applications
Common Component Architecture
All HPC Applications
9
Related Projects
ESMF Earth System Modeling Framework CCA DOE
Common Component Architecture SciDAC DOE/NSF CCSM
SciDAC Project GEMS Goddard Earth Modeling
System FMS GFDL Flexible Modeling System SWMF
Space Weather Modeling Framework WRF Weather
Research and Forecast Model CCSM Community
Climate System Model PRISM Program for Int.
Earth System Modeling
CCA
SciDAC
PRISM
WRF
CCSM
ESMF
GEMS
FMS
SWMF
10
ESMF and CCA

• Common Component Architecture (CCA) is creating a
  minimal interface and sets of tools for linking
  high performance components
• CCA can be used to implement frameworks and
  standards developed in specific domains (such as
  ESMF).
• DOE funded through SciDAC
• Collaborators include LANL, ANL, LLNL, ORNL,
  Sandia, University of Tennessee, and many more

11
ESMF and CCA, cont.

• Shujia Zhou / Arlindo da Silva (NASA GSFC)
  currently prototyping an ESMF API using CCA tool
  CCAFFEINE working towards CAM/DAO PSAS coupling
  using ESMF and CCA
• Ongoing ESMF collaboration with CCA/LANL on
  language interoperability
• Joint CCA / ESMF COMTESS meeting January 2003
• Nancy Collins (NCAR SCD) isESMF liaison to CCA

Prototype ESMF / CCA interface in CCAFFEINE
display
12
ESMF and PRISM

• A European Earth system modeling infrastructure
  project, started December 2001
• Funded by the European Commission (4.8M)
• Involves current state-of-the-art atmosphere,
  ocean, sea-ice, atmospheric chemistry,
  land-surface and ocean-biogeochemistry models
• 22 partners leading climate researchers and
  computer vendors, includes MPI, KNMI, UK Met
  Office, CERFACS, ECMWF, DMI.

13
ESMF and PRISM, cont.

• Working together to supplement CF convention for
  physical field names and quantities
• Component interface database developed by ESMF
  stores fields in model import and export states
• V. Balaji (GFDL) is ESMF liaison to PRISM

14
Outline

• ESMF Project Overview
• Related Projects and Interactions
• ESMF Design Overview
• ESMF Design Principles
• Timeline and Status

15
Architecture
Coupling Layer
ESMF Superstructure
Model Layer
User Code
Fields and Grids Layer
ESMF Infrastructure
Low Level Utilities
External Libraries
BLAS, MPI, NetCDF,
16
ESMF Components

• ESMF provides an environment for assembling
  geophysical components into an application.

Application Component
Gridded Components
Coupler Components

• ESMF provides a toolkit that components use to
• increase interoperability
• improve performance portability
• abstract common services

Component Init(), Run(), Checkpoint()
Field Halo(), Import(), Export() I/O
Grid Regrid(), Transpose() Metrics
Layout, PE List, Machine Model
17
General Features

• ESMF will be usable by models written in
  F90/C/C
• ESMF will be usable by models requiring adjoint
  capability
• ESMF will be usable by models requiring shared or
  distributed memory parallelism semantics
• ESMF will support SPMD and MPMD coupling
• ESMF will support several I/O formats, including
  GRIB/BUFR, netCDF, HDF
• ESMF will have uniform syntax across platforms
• ESMF will target a broad range of platforms, from
  major centers ? commodity hardware

18
Outline

• ESMF Project Overview
• Related Projects and Interactions
• ESMF Design Overview
• ESMF Design Principles
• Timeline and Status

19
Design PrinciplesScalable Applications
Since each ESMF application is also a component,
entire ESMF applications may be treated as
gridded components and nested within larger
applications.
climate_comp
Example atmospheric application containing
multiple coupled components within a larger
climate application
ocn2atm_coupler
ocn_comp
atm_comp
phys2dyn_coupler
atm_phys
atm_dyn
PE
20
Design PrinciplesLocal Communication
All inter-component communication within ESMF is
local.
climate_comp
This meansCoupler components must be defined
on the union of the PEs of all the components
that they couple. In this example, in order to
send data from the ocean component to the
atmosphere, the coupler mediates the send.
atm2ocn _coupler
ocn_comp
atm_comp
phys2dyn_coupler
atm_phys
atm_dyn
PE
21
Design PrinciplesModularity
Gridded Components dont have access to the
internals of other Gridded Components. Gridded
Components may - pass their States through their
argument list or - receive methods, through their
argument list, for interacting with other
Components (Transforms).
call ESMF_CompRun(atm, xform) call
ESMF_CompRun(ocn, xform)
coupler
transform
ocn_component call ESMF_StateXform(ex_state,
xform)
atm_component call ESMF_StateXform(xform,
im_state)
22
Design PrinciplesUniform Communication API
The same programming interface is used for shared
memory, distributed memory, and combinations of
both. Machine model abstraction of machine
architecture (num_nodes, num_pes_per_node) DE a
decomposition element - may be virtual, thread or
MPI process Layout an arrangement of DEs, in
which dimensions requiring faster communication
may be specified and resources arranged
accordingly.
A 2 x 6 Layout of 4 3-processor nodes
The data in a Grid is decomposed into the number
of chunks specified in the Layout.
23
ESMF Class Structure
Component
State
Superstructure
Infrastructure
Bundle
Regrid
Field
Grid
PhysGrid
DistGrid
F90
C
Layout
Array
Comm
Route
To do More work Prototype exists
MachineModel
Utilities TimeMgr, LogErr, I/O etc.
Data
Communications
24
Outline

• ESMF Project Overview
• Related Projects and Interactions
• ESMF Design Overview
• ESMF Design Principles
• Timeline and Status

25
Timeline
26
Development Status

• 8 FTEs, half in NCAR SCD, actively developing
  code
• System tests started in early January 2003
• Newly updated ESMF Architecture Document, ESMF
  Requirements Document
• Build and Test plans collected from all
  collaborators as well as core team
• Continually updated Developers Guide with coding
  conventions, configuration management, QA
  procedures, more
• Pseudocode application-level interfaces developed
  by CCSM, NSIPP, NCEP, NASA DAO
• Documents, system tests, pseudocodes on-line, on
  ESMF website

27
May 2003 Release

• Focus for May 2003 ESMF release is on
  developing sufficient infrastructure and
  superstructure to achieve the initial set of
  interoperability experiments.
• These are
• FMS B-grid atmosphere coupled to MITgcm ocean
• CAM atmosphere coupled to NCEP analysis
• NSIPP atmosphere coupled to DAO analysis

28
More information

• ESMF website http//www.esmf.ucar.edu
• On the website
• ESMF documents
• ESMF browsable source code repository
• On-line forms for community input and catalogued
  responses
• Milestone schedule, project archives and history
• Related projects
• Management plan, board and team contacts
• More