Earth System Modeling Framework

1
Earth System Modeling Framework
Cecelia DeLuca, Don Stark, Chris Hill Arctic
System Model Workshop May 20, 2008
2
Outline

• About ESMF
• Best Practices, On and Off List
• Governance
• Personal History
• Conclusion

3
About ESMF

• ESMF provides component wrappers with standard
  interfaces, a set of data structures for data
  exchanges, and customizable drivers.
• ESMF provides common utilities, such as data
  communications, regridding, time management,
  configuration, and message logging.
• Goals are component interoperability and software
  reuse.

4
ESMF Application Example
GEOS-5 Atmospheric General Circulation
Model Application Example

• Each box is an ESMF component
• ESMF State objects carry data between components
• Every component (including couplers) has a
  standard interface to facilitate exchanges
• The ESMF architecture enables the assembly of
  many different systems

5
Standardization of Component APIs

• There are only three ESMF component methods
  Initialize, Run, and Finalize (I/R/F)
• Users create a Component by assigning their user
  code I/R/F methods to an ESMF Component type
• The ESMF Component calls back into the specific
  user-assigned methods
• I/R/F methods cascade down the tree
• Small set of standard arguments

call ESMF_CompRun (myComp, importState,
exportState, clock, phase, blockingFlag, rc)
6
Standardization of Data Structures

• ESMF State data structures contain all data
  exchanged between components
• Flexibility in data representation - States can
  contain lists of varied data structures,
  including Arrays, ArrayBundles, Fields,
  FieldBundles, and other States
• The ESMF philosophy is to constrain the number
  of component methods and their arguments, but
  retain flexibility in the data structures used in
  inter-component exchanges

7
ESMF Adoption - PARSSSE

• Prepare user code
• Split user code into initialize, run and finalize
  methods and decide on components and coupling
  fields and control flow
• Adapt data structures
• Wrap native data structures either as
• ESMF Arrays (represent data in index space
  only)Regrid via user-supplied weights sparse
  matrix multiply
• ESMF Field objects (Arrays coordinates and
  metadata) Regrid via on-line weight generation
• Pack Arrays or Fields into States.Wrap time
  information in ESMF Clocks.
• Register user methods
• Attach user code initialize, run and finalize
  methods to ESMF Components through a registration
  process
• Schedule, Synchronize, and Send data between
  components
• Write couplers using ESMF redistribution, sparse
  matrix multiply, regridding, or user-specified
  transformations
• Execute the application
• Run components using an ESMF driver

8
ESMF Earth Science Components Models
9
ESMF Timeline

• First phase started in 2002 with NASA funding
• Second phase began in 2005 with a transition to
  multi-agency support and management NASA, NOAA,
  NSF, and DoD sponsors
• ESMF component interfaces are the basis of
  programs at multiple agencies
• DoD Battlespace Environments Institute (BEI)
• NOAA National Environmental Modeling System
  (NEMS)
• NASA Modeling Analysis and Prediction Program for
  Climate Variability and Change (MAP)
• Community Sediment Transport Model (ONR), others
• In operational use since August 2006, initially
  at the National Weather Service, ongoing
  transition of other ESMF coupled systems to
  operations
• Scope of current work will be completed in 2010
• Ongoing diversified funding expected

10
Release Path
2002 2003 2004 2005
2006 2007 2008 2009
2010
ESMF v1 Prototype
Full system prototype

ESMF v2 Components, VM and Utils ESMF_GridCompRun(
)
Building bottom-up
ESMF v3 Index Space Operations ESMF_ArraySparseMat
Mul()
ESMF v4 Grid Operations ESMF_GridCreate() ESMF_Fie
ldRegrid()
ESMFv5 Standardization Build, init, data types,
error handling,
Standardization
Public release ESMF v3.1.0r
11
ESMF Package Description, the Basics

• Open source, open development (browsable
  repository)
• Components can run concurrently, sequentially or
  in mixed mode
• Serial or parallel
• Single or multiple executable or combinations
• Shared or distributed memory or hybrid
• Support for model ensembles, including execution
  of multiple instances in the same address space
• 2000 unit tests, system tests, and examples
  regression tested nightly on 26
  platform/compiler combinations
• Reference Manual, Users Guide and the examples
  therein updated automatically with code changes

12
ESMF Package Description, Coupling

• Data transformations can be executed within a
  coupler component, or arranged in a coupler
  component and executed directly between model
  components
• Coupling can be done in index space or physical
  space
• Performance lt 5 overhead in time to solution
  vs customized native approaches, highly scalable
  in performance and memory (performance reports
  online)

13
Upcoming Releases

• Public Release 3.1.0r, May 2008
• Main capabilities added since last public release
  - 2.2.2rp3, December 2006
• General grid and data representation and parallel
  regridding support through an Array index space
  layer (user supplied interpolation weights)
  logically rectangular grids, multi-patch grids,
  unstructured grids
• Representation of rectilinear and curvilinear
  grids in a Grid class (preparation for
  framework-generated interpolation weights)
• New ports including gfortran and Solaris
• Compact, simpler interface cut number of public
  data handling methods in half (304 -gt 147, with
  caveats)

14
Upcoming Releases (cont.)

• Internal Release 3.1.1, June 2008
• Initial implementation of observational data
  streams
• Generation of regridding interpolation weights
  for logically rectangular grids, bilinear and
  higher order method
• Attribute class can store standardized metadata
  packets and represent metadata hierarchies -
  e.g., State metadata includes the metadata of
  Fields that are stored in it
• Initial implementation of unstructured grid
  interface
• Patch Release 3.1.0rp, Summer 2008
• Petascale optimizations for Array and ArrayBundle
  sparse matrix multiply

15
Relationship to CCSM

• Passed CCSM Stage 1 evaluation of ESMF in May
  2007, showing lt 5 overhead, comparable memory
  use, no change to component internals.
• Result creation of cpl7 based on ESMF design
• Currently working on Stage 2 evaluationCreation
  of an ESMF version of cpl7 called ESMCoupler
• Expected advantages
• Greater interoperability with other components
• More flexible architecture that doesnt rely on a
  single central coupler
• On-line regridding
• Integrated handling of metadata

16
Team Composition

• Team composition (blue is off-site)
• Manager (agency coordination and technical
  coordination)
• Operations Manager (website, metrics, space
  issues, local administration)
• Integrator/Test lead (regression testing, release
  management)
• Tester for numerical methods
• 7 developers
• Systems level/porting/build
• Low level data structures and architecture
• Structured grids
• Meshes
• High level numerical data structures
• Language interfaces
• Utilities including calendaring
• Metadata and attributes
• External and related
• External ½ FTE performance testing
• Related FTE for component distribution portal

17
Distributed Development

• GOAL - Everybody on the team has access to all
  information, current and past
• Archived email list where all development
  correspondence gets ccd
• Frequent telecons with minutes
• Web browsable repository, mail summary on
  check-ins
• Daily archived test results
• Monthly archived metrics
• Public archived trackers (bugs, feature requests,
  support requests, etc.)

18
ESMF Governance

• Multiple timescales, all staff levels
• Places for structured argument

ExecutiveManagement
Executive Board Strategic Direction Organizational
Changes Board Appointments
annually
Reporting
Interagency Working Group Stakeholder
Liaison Programmatic Assessment Feedback
Advisory Board External Projects
Coordination General Guidance Evaluation
Reporting
Working Project
Joint Specification Team Requirements
Definition Design and Code Reviews External Code
Contributions
Change Review Board Development
Priorities Release Review Approval
quarterly
Functionality Change Requests
weekly
Resource Constraints
Implementation Schedule
Collaborative Design Beta Testing
Core Development Team Project Management Software
Development Testing Maintenance Distribution
User Support
daily
19
Values

• Community driven development and community
  ownership
• Openness of project processes, management, code
  and information
• Correctness
• Commitment to a globally distributed and diverse
  development and customer base
• Simplicity
• Efficiency
• Public storage of project records and other
  information
• Engagement
• Web link for detail http//www.esmf.ucar.edu/abou
  t_us/values.shtml

20
How to Get Started Get Help

• Web meeting with the development team
• Support list esmf_support_at_ucar.edu
• Code Examples page and FAQ
• Tutorials, on-line and in person
• Annual Meeting (Fort Lauderdale, FL May 28-30,
  2008)
• Reference Manual and Users Guide

21
Governance

• Management of ESMF requires governance that
  recognizes social and cultural factors as well as
  technical factors
• Main objectives of governance
• Enabling people to argue in a civilized,
  contained, constructive way
• Enabling people to set priorities and make
  decisions based on resource realities
• Coordinating at many levels, from developer to
  center director to program manager

22
Facilitating ScienceCoupled Climate-Chemistry
with ESMF
The image shows results from a version of the
GEOS-5 atmospheric general circulation model
coupled to a stratospheric chemistry
package (STRAT-CHEM), also developed at NASA, but
independently of GEOS-5,which has now been made
ESMF compliant.