Seismic Hazard Modeling using Heterogeneous Scientific Workflows - PowerPoint PPT Presentation

1 / 139
About This Presentation
Title:

Seismic Hazard Modeling using Heterogeneous Scientific Workflows

Description:

Seismic Hazard Modeling using Heterogeneous Scientific Workflows – PowerPoint PPT presentation

Number of Views:244
Avg rating:3.0/5.0
Slides: 140
Provided by: PhilipMa3
Category:

less

Transcript and Presenter's Notes

Title: Seismic Hazard Modeling using Heterogeneous Scientific Workflows


1
Seismic Hazard Modeling using Heterogeneous
Scientific Workflows
Philip MaechlingInformation Technology
ArchitectSouthern California Earthquake Center
(SCEC)University of Southern Californiahttp//ww
w.scec.org14 September 2007
2
(No Transcript)
3
Example SCEC Project Rosters
  • Large Collaborative Projects involving computer
    scientists and geophysicists worked together.
  • TeraShake
  • Kim B. Olsen (SDSU), Bernard Minster (IGPP),
    Reagan Moore (SDSC), Steve Day (SDSU), Phil
    Maechling (USC), Tom Jordan (USC), Marcio Faerman
    (SDSC), Geoffrey Ely (IGPP), Boris Shkoller
    (IGPP), Carey Marcinkovich (EXxonMobil), Jacobo
    Bielak (CMU), David Okaya (USC), Ralph Archuleta
    (UCSB), Steve Cutchin (SDSC) , Amit Chourasia
    (SDSC), George Kremenek (SDSC), Yuanfang Hu
    (SDSC), Arun Jagatheesan (SDSC), Nancy
    Wilkins-Diehr (SDSC), Richard Moore (SDSC), Bryan
    Banister (SDSC), Leesa Brieger (SDSC), Amit
    Majumdar (SDSC), Yifeng Cui (SDSC), Giridhar
    Chukkapalli (SDSC), Qiao Xin (SDSC), Donald Thorp
    (SDSC), Patricia Kovatch (SDSC), Larry Diegel
    (SDSC), Tom Sherwin (SDSC), Christopher Jordan
    (SDSC), Marcus Thiebaux (ISI), Julio Lopez (CMU)
  • Workflow Systems Including CyberShake and
    Earthworks
  • Hans Chalupsky (ISI), Maureen Dougherty
    (USC/HPCC), Ewa Deelman (ISI), Yolanda Gil (ISI),
    Sridhar Gullapalli (ISI), Vipin Gupta (USC), Carl
    Kesselman (ISI), Jihie Kim (ISI), Gaurang Mehta
    (ISI), Brian Mendenhall (USC/HPCC), Thomas Russ
    (ISI), Gurmeet Singh (ISI), Marc Spraragen (ISI),
    Garrick Staples (USC/HPCC), Karan Vahi (ISI),
    Yifeng Cui (SDSC), Thomas Jordan (SCEC), Li Zhao
    (USC), David Okaya (USC), Robert Graves (URS),
    Ned Field (USGS), Nitin Gupta (SCEC), Scott
    Callaghan (USC), Hunter Francoeur (USC), Joanna
    Muench (IRIS), Philip Maechling (USC)

4
Outline
  • SCEC Earthquake System Science
  • Earthquake System Science Computing
  • Heterogeneous Workflow Research
  • SCEC View of the Computing World

5
One Week of Earthquakes in California
6
Seismic Hazard Analysis as a System-Level
Earthquake Research Problem
  • Definition Specification of the maximum
    intensity of shaking expected at a site during a
    fixed time interval
  • Example National seismic hazard maps
  • Intensity measure peak ground acceleration (PGA)
  • Interval 50 years
  • Probability of exceedance 2

7
(No Transcript)
8
To SCEC, a Geosystem is Defined by Predicted
Behavior
  • A geosystem comprises a set of interacting
    elements that changes in time according to a set
    of prescribed laws, and produce a specific
    beharior.
  • In other words, the behavior defines the system
  • Geosystem Behavior
  • Climate system (a.k.a. Earth system) climate
  • mantle convection plate tectonics
  • core dynamo magnetic field
  • orogen mountain building and erosion
  • active fault system earthquakes
  • super-cell thunderstorm tornado
  • volcano magmatic eruption
  • petroleum reservoir accumulation flow of
    petroleum

9
Earthquake System Science research seeks to
predict ground motions. Ground motion (future
shaking) is the behavior that the SCEC system
science program seeks to predict.
10
Puente Hills EarthquakeScenario
(E. Field, H. Seligson, N. Gupta, V. Gupta, T.
Jordan K. Campbell, 2005)
Projected losses 82 B - 252 B 3,000 - 18,000
fatalities 142,000 -735,000 displaced
households 30,000 - 99,000 tons of debris
Percent Building Loss
11
System Science Integrates Observations,
Geological Models, and Simulations
Seismicity
Paleoseismology
Geologic structure
Local site effects
Faults
Seismic Hazard Model
Stress transfer
Rupture dynamics
Crustal motion
Crustal deformation
Seismic velocity structure
12
Earthquake System Science
  • Prediction of seismic shaking is a true
    system-level problem
  • Involve nonlinear, multiscale interactions among
    many geologic components, themselves often
    complex subsystems
  • SCEC research seeks to develop a predictive
    understanding of earthquake processes by applying
    an interdisciplinary, system science, research
    approach that uses Southern California as a
    natural laboratory.

13
Foster and Kesselman - IEEE Computer (2006)
14
  • Southern California Earthquake Center
  • Involves 500 scientists at 55 institutions
    worldwide
  • Focuses on earthquake system science using
    Southern California as a natural laboratory
  • Translates basic research into practical products
    for earthquake risk reduction

Communicate understanding to society at large as
useful knowledge for reducing earthquake risk
Integrate information into a comprehensive,
physics-based understanding of earthquake
phenomena
Gather data on earthquakes in Southern
California and elsewhere
SCEC Mission Statement
15
  • Southern California Earthquake Center
  • Involves 500 scientists at 55 institutions
    worldwide
  • Focuses on earthquake system science using
    Southern California as a natural laboratory
  • Translates basic research into practical products
    for earthquake risk reduction

Tectonic Evolution B.C.s
Deformation Models
Earthquake Rupture Forecasts
Earthquake Rupture Models
Fault Models
Block Models
Seismic Hazard Products
Risk Mitigation Products
Anelastic Structures
Attenuation Relationships
Ground Motion Simulations
SCEC Master Model Focus Groups
16
SCEC Member Institutions(October 1, 2006)
17
SCEC Research Funding Opportunities
  • SCEC performs a yearly proposal process and
    distributes a large percentage of NSF and USGS
    SCEC funding to individual research groups.
  • Request for Proposals Posted on SCEC Web site
    September.
  • Proposals Due in Nov
  • Proposals Selected in Jan
  • Funding available in March
  • Proposal typically small (30k more or less).
    Must be in line with SCEC Science plan. A wide
    range of activities are funded
  • Workshops
  • Support for Interns
  • Field studies
  • Simulation programs
  • Geoinformatic developments
  • etc

18
SCEC3 Organization
SCEC Director Board of Directors
External Advisory Council
Planning Committee
CEO Program
Center Administration
Information Architect
Earthquake Geology
Unified Structural Representation
Seismic Hazard Risk Analysis
CME
Tectonic Geodesy
Fault Rupture Mechanics
PetaSHA
Knowledge Transfer
Seismology
Crustal Deformation Modeling
Public Outreach
PetaShake
Lithospheric Architecture Dynamics
K-12 Informal Education
CSEP
Earthquake Forecasting Predictability
USEIT/SURE Intern Programs
ExGM
Ground Motion Prediction
MPRESS
ACCESS Forum
Disciplinary Committees
CEO Activities
Special Projects
Focus Groups
19
  • SCEC Community Modeling Environment (SCEC/CME)
  • Funded by NSF/EAR/CISE
  • 10.0M for 5 years 1 year ext. (ending 2007)
  • Science thrust areas
  • Extend scale range of deterministic ground-motion
    simulations and dynamic rupture simulations
  • Compute physics-based PSHA maps and validate them
    using seismic and paleoseismic data
  • Computer science objectives
  • Grid-based Workflows
  • Data Management
  • Knowledge Capture
  • Promote vertical integration of
    cyberinfrastructure

CME
PetaSHA
PetaShake
CSEP
ExGM
MPRESS
20
  • A Petascale Cyberfacility for Physics-Based
    Seismic Hazard Analysis (PetaSHA)
  • Funded by NSF/EAR
  • 2.1M for 2 years (ending 2008)
  • Science thrust areas
  • Extend scale range of deterministic ground-motion
    simulations and dynamic rupture simulations
  • Compute physics-based PSHA maps and validate them
    using seismic and paleoseismic data
  • Computer science objectives
  • Reach petascale computing capability by 2009
  • Promote vertical integration of
    cyberinfrastructure

CME
PetaSHA
PetaShake
CSEP
ExGM
MPRESS
21
  • Enabling Earthquake System Science through
    Petascale Calculations (PetaShake)
  • Funded by NSF/EAR/CISE/OCI
  • 1.8M for 2 years (ending 2009)
  • Science thrust areas
  • Extend scale range of deterministic ground-motion
    simulations and dynamic rupture simulations
  • Highly scalable Earthquake Wave Propagation codes
  • Computer science objectives
  • Codes that scale to 100,000 processors by 2009
  • Workflow-based On-demand verification and
    Validation for Petascale codes

CME
PetaSHA
PetaShake
CSEP
ExGM
MPRESS
22
  • Collaboratory for the Study of Earthquake
    Predictability (CSEP)
  • Funded by W. M. Keck Foundation
  • 1.2M for 3 years (ending 2009)
  • Science thrust areas
  • Formulate Techniques for Evaluating Earthquake
    Predictions
  • Develop international collaboration to study
    earthquake predictability
  • Computer science objectives
  • Robust, reliable testing center with reproducible
    results
  • Automated executing of prediction models and
    prediction evaluation algorithms

CME
PetaSHA
PetaShake
CSEP
ExGM
MPRESS
23
Students in Computer Science and Geoscience
SCEC/USEIT
2003
2002
2004
2005
24
(No Transcript)
25
Current SCEC ACCESS Students
  • SCEC - Advancement of Cyberinfrastructure Careers
    through Earthquake System Science (ACCESS)
    involves both undergraduate and graduate students
    in computer science and geoscience research

26
  • Scientific Publications and Public Outreach raise
    awareness and improve preparedness.

27
Outline
  • SCEC Earthquake System Science
  • Earthquake System Science Computing
  • Heterogeneous Workflow Research
  • SCEC View of the Computing World

28
SCEC Science Goals
  • SCEC seeks to improve predictive earthquake
    system models by constantly integrating more
    realistic physics into the numerical simulations.
  • As simulations become more physically realistic
    the computational and data management
    requirements increase enormously

29
Types of SCEC Numerical Simulations
Earthquake Wave Propagation Simulations
Ensemble Probabilistic Seismic Hazard Calculations
Friction-based Dynamic Rupture Simulations
30
Types of SCEC Physics-based Earthquake Simulations
  • Large Earthquake Wave Propagation Simulations
  • Dynamic Earthquake Rupture Simulations
  • Ensemble Probabilistic Seismic Hazard Simulations
  • Data Inversion Simulations improve Structure
    Models
  • Ground motion coupled to building response

31
System Science Implies Broad Computing Needs
SCEC Cyberinfrastructure Needs
  • High Performance Computing (Capability Computing)
  • Distributed, high throughput, grid computing
    (Capacity Computing)
  • Large-scale Data and Metadata Management (Data
    Intensive Computing)
  • 2D, 3D, and 4D Data Visualization
  • Scientific workflows
  • Interactive Science Gateway technologies and
    Collaboration tools.
  • High Performance Computing Expertise.
  • Urgent Computing (after a significant event)

32
SCEC Simulation Working Group
33
(No Transcript)
34
Development of SCEC Community Models Structural
Representation
35
SCEC Community Fault Model
A. Plesch and J. Shaw (2003)
36
SCEC Community Velocity Model
H. Magistrale et al. (2000)
37
SCEC Community Block Model
Set of interconnected, closed volumes that are
bounded by major faults, as well as topography,
base-of-seismicity, and Moho surfaces.
Intended for use in fault systems analysis
(FEM) property modeling
J. Shaw et al. (2004)
38
SCEC Crustal Motion Map From GPS Data
CMM.3.0.1 (Agnew et al., 2003)
39
Unified Structural Representation
Crustal Motion Map
Tectonic models
Community Fault Model
Community Block Model
Structural models
40
SCEC Science Goals
  • SCEC seeks to improve predictive earthquake
    system models by constantly integrating more
    realistic physics into the numerical simulations.
  • Example 1 Ground motion predictions based on
    wave propagation simulations through realistic
    geological structures are more physically
    realistic than ground motion predictions using
    attenuation relationships and simple geological
    models.

41
Predicted Strong Ground Motion Predictions for
Hypothetical Earthquake Old Style (Attenuation
Relationship)
Light
Moderate
Heavy
Very Heavy
Ground Motion Levels for a Hypothetical Ml 6.75
in Hollywood
42
Wave Propagation Assuming Simple Consistent
Geology
43
Geology in Los Angeles Is Complex with Deep
Sedimentary Basins
44
Wave Propagation using 3D waveform modeling and
Realistic Geology in Los Angeles
45
Verification of Earthquake Wave Propagation Codes
Using Simplified Problem
Mxy
46
SCECs Predictive simulations are validated Using
Historical Earthquakes.We perform simulations
of historic earthquakes for which there are
seismic recordings.These two maps compare
observed ground motions (top) to results of
simulation (bottom) for a recent earthquake
47
SCECs Predictive Models Are Validated Using
Historical Earthquakes
48
Scenario Earthquake Simulations Puente Hills
Peak SA 2.0 magnitude Map
Velocity Y Component Animation
Puente Hills Simulation Scenario Earthquake (10
Hz) Robert Graves (AWM), Amit Chourasia et al
(Viz)
49
TeraShake-1 Simulation Results New Scale and
Resolution (1.2GB Mesh Points)
50
SCEC Science Goals
  • SCEC seeks to improve predictive eathquake system
    models by constantly integrating more realistic
    physics into the numerical simulations.
  • Example 2 Fault rupture simulations that are
    constrained to obey friction laws (dynamic
    ruptures) are more physically realistic than
    ruptures simulations that arent constrained by
    friction laws (kinematic ruptures).

51
Cartoons that illustrate kinematic and dynamic
ruptures
52
Kinematic Rupture Simulation Using a Constant
Rupture Velocity
Why Faults Start Slipping and Why They Stop
Slipping is Still Difficult to Understand and
Model.
53
(No Transcript)
54
Earthquake Rupture Mode Code Verification
  • Supported by SCEC ESP Focus Group
  • First workshop held in November, 2003
  • Second workshop planned for Summer, 2004
  • Results will be archived in CME

55
TeraShake-2 Simulation Results
56
SCEC Science Goals
  • SCEC seeks to improve predictive earthquake
    system models by constantly integrating more
    realistic physics into the numerical simulations.
  • Example 3 Produce probabilistic seismic hazard
    maps by running ensemble calculations that
    simulate all possible earthquakes that might
    affect the region.

57
Probabilistic Seismic Hazard Analysis Long Term
Ground Motion Forecast
  • Definition Specification of the maximum
    intensity of shaking expected at a site during a
    fixed time interval
  • Example National seismic hazard maps Used as
    basis for Building Codes
  • Intensity measure peak ground acceleration (PGA)
  • Interval 50 years
  • Probability of exceedance 2

58
Probabilistic Seismic Hazard Calculations Require
Ensemble Calculations (like weather forecasts)
59
Probabilistic Seismic Hazard Calculations Require
Ensemble Calculations (like weather forecasts)
60
The multiple programs that are used to calculate
a CyberShake hazard curve calls for use of
scientific workflow technology.
61
  • Simulate all possible earthquakes within 200KM
    of site we are studying.

62
For Every Possible Earthquake, Define Every
Possible Rupture
63
Validation Using Precarious Rocks
UNR Database
64
Outline
  • SCEC Earthquake System Science
  • Earthquake System Science Computing
  • Heterogeneous Workflow Research
  • SCEC View of the Computing World

65
SCEC IT Priorities
  • SCEC IT groups charge is to perform large scale
    scientific computing
  • We must get research done
  • We tend to be technology neutral
  • We are careful to apply technology only when it
    helps us get our research done.

66
SCEC Research-oriented computing needs are
heterogeneous - full of contrasts
  • Compiled codes (C, Fortran)
  • Interpreted code (Java, Python)
  • Geological Time scales
  • Emergency Response Time scales
  • Distributed Stateful Object codes.
  • Stateless web services.
  • Supercomputer savvy software engineers.
  • Domain experts with little high performance
    computing experience.
  • Very Long running jobs (24hours)
  • Very short running jobs (lt 1 minute)
  • Small serial calculations
  • Very large parallel calculations
  • Single (Hero) runs - Parallel supercomputers
    calculations
  • Many small calculations high throughput jobs
  • Very large data files (1TB)
  • Many small files (1M)
  • Data in relational databases
  • Legacy, heavily used codes
  • Rapidly changing codes
  • Single programs
  • Multi-stage processing streams (workflows)

67
Accepting Heterogeneity
  • We have been forced into the view that
    heterogeneity is inevitable and we must deal with
    it.
  • Grids help deal with heterogeneity.
  • Our workflow tools are specifically selected to
    support grid-based workflows in a heterogeneous
    environment.

68
Some Characteristics of Grids
Numerousresources
Owned by multiple organizations individuals
Connected by heterogeneous, multi-level networks
Different security requirements policies
Different resource management policies
Geographically distributed
Unreliable resources and environments
Resources are heterogeneous
Slide by Hiro
69
Dealing with Heterogeneity of Computing
  • Applying grid software stack to homogenous
    computing environment may be wasteful.
  • Example Originally, TeraGrid was envisioned as
    homogenous hardware and software.
  • Example of Unclear on the concept of grids as
    tools to deal with heterogeneity.
  • It has now evolved away from this.

70
Example SCEC Research Application OpenSHA
  • Research Computing Challenge
  • Collection of existing scientific codes that
    implement different techniques for calculating
    existing probabilistic seismic hazards curves and
    maps.
  • Mostly small, short, serial calculations.
  • Must be extensible as new techniques and codes
    are developed.
  • Solution
  • Standard object model for domain
  • Java language framework
  • Java interface wrapping FORTRAN codes to avoid
    re-writing existing solutions.
  • Distributed, stateful objects implemented using
    Java RMI
  • CondorG-based calculations for high throughput
    codes.

71
Pathway 1 OpenSHA
Time Span
OpenSHA A Community Modeling Environment
for Seismic Hazard Analysis
Earthquake- Rupture Forecast
IM
Rupn,i
Site
Type, Level
Sourcei
Intensity-Measure Relationship
72
OpenSHA
A framework where any arbitrarily complex (e.g.,
physics based) SHA component can plug in for
end-to-end SHA calculations.
  • open source
  • object oriented
  • platform ind.
  • web/GUI enabled
  • distributed (potentially)
  • Java (or wrapped code)
  • validated

73
OpenSHA
Applications Available
1) Hazard Curve Calculator
Publications WGCEP-2002 ERF (Field et al. 2005,
SRL) Distributed Object Technologies (Maechling
et al., 2005, SRL)
74
OpenSHA
Applications Available
1) Hazard Curve Calculator
2) Scenario ShakeMap Calculator
75
OpenSHA
Applications Available
1) Hazard Curve Calculator
2) Scenario ShakeMap Calculator
3) Hazard Map Data Calculator
4) Hazard Map Plotter
Utilization of GRID computing (Maechling et al.,
2005, SRL)
76
OpenSHA Publications
Field, E.H., H.A Seligson, N. Gupta, V. Gupta,
T.H. Jordan, and K.W. Campbell (2005). Loss
Estimates for a Puente Hills Blind-Thrust
Earthquake in Los Angeles, California, Earthquake
Spectra, 21, 329-338. Field, E.H., N. Gupta, V.
Gupta, M. Blanpied, P. Maechling, and T.H. Jordan
(2005). Hazard Calculations for the WGCEP-2002
Earthquake Forecast Using Distributed Object
Technologies, Seism. Res. Lett., 76,
161-167. Maechling, P., V. Gupta, N. Gupta, E.H.
Field, D. Okaya and T.H. Jordan. (2005) Seismic
Hazard Analysis Using Distributed Computing in
the SCEC Community Modeling Environment, Seism.
Res. Lett., 76, 177-181. Field, E.H., V. Gupta,
N. Gupta, P. Maechling, and T.H. Jordan (2005).
Hazard Map Calculations Using OpenSHA and GRID
Computing, Seism. Res. Lett., In
Press. Maechling, P., V. Gupta, N. Gupta, E.H.
Field, D. Okaya and T.H. Jordan. (2005). Grid
Computing in the SCEC Community Modeling
Environment , Seism. Res. Lett., In Press.
77
Example SCEC Research Application Community
Modeling Environment to Help Non-expert Users Run
Wave Propagation Simulations
  • Research Computing Challenge
  • Help user setup and run earthquake wave
    propagation simulations
  • Run simulations on any available supercomputer
  • Run multiple stages including setup, execution,
    data extraction, and discovery
  • Manage data and provide data discovery.
  • Solution
  • Formulate calculations as a grid-based workflow.
  • Build knowledge-based users interfaces for
    workflow construction and data discovery.
  • Utilize USC High Performance Computing and
    TeraGrid computers
  • Integrate Data Management using Digital Library
    technology (Storage Resource Broker)

78
SCEC/CME Computational Pathway Construction
A major SCEC/CME objective is the ability to
construct and run complex computational pathways
for SHA
Lat/Long/Amp (xyz file) with 3000 datapoints
(100Kb)
Define Scenario Earthquake
ERF Definition
Calculate Hazard Curves
Extract IMR Value
Plot Hazard Map
9000 Hazard Curve files (9000 x 0.5 Mb 4.5Gb)
IMR Definition
GMT Map Configuration Parameters
Gridded Region Definition
Probability of Exceedence and IMR Definition
Pathway 1 example
79
Users run Wave Propagation Simulations
  • Our simulation system used a lifecycle concept
    that recognized several phases in workflow
    development and use and tried to address each one
  • Construct workflow.
  • Submit workflows
  • Monitor workflows
  • Archive workflow data
  • Discover workflow data
  • Retrieve workflow data

80
(No Transcript)
81
Creation of Workflows in Layers of Increasing
Detail
  • Workflow Template
  • Specifies executables and dataflow
  • No data specified, just their type
  • Workflow Instance
  • Specifies data files for a given template
  • Logical file names, not physical file replicas
  • Executable Workflow
  • Specifies physical locations of data files,
    hosts/pools for execution of jobs, and data
    movement jobs

82
The Process of Creating an Executable Workflow
User guided
  • Creating a valid workflow template (human guided)
  • Selecting application components and connecting
    inputs and outputs
  • Adding other steps for data conversions/transforma
    tions
  • Creating instantiated workflow
  • Providing input data to pathway inputs (logical
    assignments)
  • Creating executable workflow (automatically)
  • Given requirements of each model, find and assign
    adequate resources for each model
  • Select physical locations for logical names
  • Include data movement steps, including data
    deposition steps

Automated
83
USC HPCC Resources
Grid
SCEC Computing Resources
TeraGrid Computing Resources
84
Common TeraGrid Software Stack a Key Component of
a Scientific Computing Environment
85
SCEC Workflow System Software Stack
Pegasus Meta-scheduler Ability converting
abstract workflows to concrete workflows Automatic
expansion of workflows to stage and transfer
files. Virtual Data Toolkit (VDT) Includes
Condor, Globus Job scheduling including condor
glide-in provisioning for embarrisingly parallel
execution on cluster Metadata Catalog
Service Metadata archive and data
discovery Replica Location Service Mapping
logical files to physical files Globus GSI
security supported at USC and TeraGrid. GridFTP
for data transfer. Support for Condor job
scheduling. Storage Resource Broker Client (SRB
Server at SDSC).
86
INTEGRATED WORKFLOW ARCHITECTURE
J. Zechar _at_ USC (Teamwork Geo CS)
Workflow Template Editor (CAT)
Query for components
D. Okaya _at_ USC
Tools
Domain Ontology
Workflow Template (WT)
Workflow Library
Component Library
Query for WT
Data Selection
Query for data given metadata
L. Hearn _at_ UBC
COMPONENTS
I/O data descriptions
Conceptual Data Query Engine (DataFinder)
Metadata Catalog
Workflow Instance (WI)
Execution requirements
Engineer
Workflow Mapping (Pegasus)
Grid info svcs
Tools
Grid
K. Olsen _at_ SDSU
Executable Workflow
87
Assisted Pathway Composition
  • Problem In order to bring sophisticated models
    to a wide range of users we need to provide
    assistance and automation while allowing users to
    guide the process
  • Approach Mixed-initiative system that helps
    users create, reuse, and combine pathways by
    exploiting
  • Knowledge-based descriptions of components
  • Ontology of components and component types based
    on common features and parameter constraints
  • Analysis of (partially constructed) pathways
    based on AI planning techniques
  • Provide formal definitions of desirable
    properties of pathways

ErrorScan Input Workflow W ltC,L,I,Ggt Output
list of errors and corresponding fix
suggestions I. If W is not purposeful, return
Error. Suggestions define end result e using
types from the KB, AddEndResult (e). . II.
For each Component C in W a. If C is not
Justified, return Error. Suggestions ?p
that is output-parameter (c), find components
cj in the workflow or the KB that have
pj as input- parameter(cj), and subsumes(pj,p),
AddLink(c,p,cj,pj) b. If C is not
grounded, return Error. Suggestions (?
Cj ? FindDirectSubtypes(c),
SpecializeComponent(C, Cj). c. For each i in
input-parameter(c) 1. If i is not
Satisfied, return Error.
Suggestions ? cj ? C with output parameter pj
such that
subsumes(range(c,i),range(cj,pj))
AddLink(cj,pj,c,i).
Suggestions ? cj ? FindMatchingOutput (i)),
AddLink(cj,pj,c,i).
SuggestionAddAndLinkComponent (W,
AddInitialInput(i),range( i), c, i) III.
For each Link L in W a.If L is not
Consistent, return Error. Suggestions ?
Ci ? FindInterPosingComponent(L),
InterposeComponent (Ci, L).
Suggestion RemoveLink(L). b. If L is
Redundant, return Error. Suggestion
RemoveLink (L).
88
CAT Composition Analysis Tool Now Called Wings
Declarative descriptions of models are linked to
ontologies and reasoners
System reasons about model constraints and points
out errors and fixes
User builds a pathway specification from library
of models
System guarantees correctness of pathway templates
89
SCEC Digital Library Contains TB of Data and
Metadata
  • SRB-based Digital Library
  • More than 160 Terabytes of tape archive
  • 4 Terabytes of on-line disk
  • 5 Terabytes of disk cache for derivations

90
Conceptual Queries DataFinder
  • Browsing interface is the wrong metaphor
  • Too many files to look through
  • Too many metadata atttributes
  • Search is the right answer
  • Focus on items of interest
  • Provide conceptual level abstractions
  • Need scalable approach

91
Velocity Mesh Conceptual Model
92
Maechling, P., H. Chalupsky, M. Dougherty, E.
Deelman, Y. Gil, S. Gullapalli, V. Gupta, C.
Kesselman, J. Kim, G. Mehta, B. Mendenhall, T.
Russ, G. Singh, M. Spraragen, G. Staples, K. Vahi
(2005) Simplifying Construction of Complex
Workflows for Non-Expert Users of the Southern
California Earthquake Center Community Modeling
Environment, ACM SIGMOD Special issue on
Scientific Workflows, Record Vol. 34 No. 3,
September 2005, pp. 24-30
93
Lesson from Our Integrated Workflow Architecture
  • Conclusions of Paper (maybe still valid)
  • Move knowledge about the workflow components
    (programs) and computational infrastructure
    (grid) into knowledge-bases, databases
  • Let experts in the domain, or in the computing
    environment, create these data structure.
  • Then the workflow tools can utilize these data
    structures and the user doesnt need to be an
    expert in these area.

94
Lesson from Our Integrated Workflow Architecture
  • On Workflows
  • The separation into 3 stages of creation
    (template, abstract, and concrete) works well. It
    is especially well suited for grid-based
    workflows.
  • Many tools, especially visual workflow
    construction tools, combine the three stages, and
    they only help you create an instance of a
    concrete workflow. This makes it difficult to
    run a lot of workflows through a GUI.
  • Achilles heel of workflow tools continues to be
    workflow template construction (First stage of
    workflows). Workflow templates are difficult to
    create and modify

95
Lesson from Our Integrated Workflow Architecture
  • On Knowledge Representation
  • Work started on this workflow system continues
  • "Provenance Trails in the Wings/Pegasus Workflow
    System", Jihie Kim, Ewa Deelman, Yolanda Gil,
    Gaurang Mehta, Varun Ratnakar. In To appear in
    Concurrency and Computation Practice and
    Experience, Special Issue on the First Provenance
    Challenge, 2007
  • "Wings for Pegasus Creating Large-Scale
    Scientific Applications Using Semantic
    Representations of Computational Workflows",
    Yolanda Gil, Varun Ratnakar, Ewa Deelman, Gaurang
    Mehta, Jihie Kim. In To appear in Proceedings of
    the 19th Annual Conference on Innovative
    Applications of Artificial Intelligence (IAAI),
    Vancouver, British Columbia, Canada, July 22-26,
    2007
  • "Workflow Composition", Yolanda Gil. In In
    Workflows for e-Science, D. Gannon, E. Deelman,
    M. Shields, I. Taylor (Eds), Springer Verlag,
    2006
  • The workflow construction tools that use
    knowledge-bases require knowledge-engineers. C
    programmers are had enough to find and hire, and
    knowledge engineers are much harder. So,
    knowledge-bases were very difficult to maintain.
  • Development of ontologies about computer domains
    worked because computer terms were less ambigous.
    Development of scientific ontologies was rough
    sledding and was not providing enough benefit
    helping us get our SCEC research done enough to
    continue to support.

96
Integrated Workflow system Evolved into Science
Gateway
  • Goals of Earthworks System
  • Automatically run simulation after local event as
    an urgent workflow.
  • Post results (data products) on SCEC web site.
  • Allow scientists to re-configure and rrun
    simulations.

97
Integrated Workflow System Evolved into SCEC
Earthworks Science Gateway
wave.usc.edu
ia64.sdsc.edu
scecdata.usc.edu
desktop
intensity.usc.edu
Maechling, P., J. Muench, H. Francoeur, D. Okaya,
Y. Cui (2007) SCEC Earthworks Science Gateway
Interactive Configuration and Automated Execution
of Earthquake Simulations on the TeraGrid, In
Proceeding of TeraGrid 2007, Madison Wisconsin
98
Simulation Configuration Interface
underlying XML-based description file - no web
page upkeep
99
SCEC Earthworks Science Gateway Widening Access
to the TeraGrid
Velocity Model
SeismicWave Prop.
Intensity measures (PGA-PGV
spectral accel.) wave processing (debias,
filter, vel2accel.). PEER
deconvol. archive (dig. libr.) PG map
making. Viz SDSC Viz ISI
CVM 4.0 CVM 3.0 CVM 2.2 Harvard VM. 1D
Hadley- Kanamori. Constant
velocity Top layer halfspace.
TeraShake2 _at_HPCC.usc TeraShake2
_at_Teragrid.sdsc Graves FD _at_HPCC.usc Graves FD
_at_Teragrid.sdsc Carnegie-Mellon FE Caltech
spectral element
100
Choices in AWM code and Velocity Model
Same velocity model (CVM3.0) three AWM codes
Same AWM code (Olsen) three velocity models
Hollywood EQ -118.398o, 34.053o 7.00 km depth Mw
4.23, strike165o dip60o, rake0o Region 24 x
24 x 12 km dx 150 m
101
Earthworks Gateway Supports On-demand
verification and validation A emphasis for us.
  • Research Computing Challenge
  • Keeping codes validated as they change
  • Verification and validation is a multi-step
    process.
  • Solution
  • Implement verification problems as workflows

102
Example PEER UHS-1 Verification
Lifelines Program Task 1A01
Mxy
103
Example SCEC Research Application Ensemble
Calculations using both MPI and Post processing
  • Research Computing Challenge
  • Run large MPI jobs that output large data files
  • Run 100K serial post processing jobs
  • Manage data and provide data discovery.
  • Solution
  • Formulate calculations as a grid-based workflow.
  • Utilized clusters as condor pools
  • Use condor glide-ins to reduce job scheduler
    delays
  • Utilize USC High Performance Computing and
    TeraGrid computers
  • Archive results and metadata in database

104
Physics-based Probabilistic Seismic Hazard
Calculations - CyberShake Platform
  • Objective
  • Capability for physics-based probabilistic
    seismic hazard calculation in Southern
    California, accounting for source complexity and
    3D earth structure
  • Simulates ground motions for potential fault
    ruptures within 200 km of each site
  • 12,700 sources in Southern California from USGS
    2002 earthquake source model
  • Extends USGS 2002 to multiple hypocenters and
    slip models for each source
  • 100,000 ground motion simulations for each site

105
Workflow used only on high throughput aspect of
the calculations
The multiple programs that are used to calculate
a CyberShake hazard curve calls for use of
scientific workflow technology.
106
SCEC workflows on the TeraGrid Described in
Workflows for e-Science
Executable workflow
Maechling P., E. Deelman, G. Mehta, R. Graves, L.
Zhao, N. Gupta (2007) SCEC CyberShake Workflows -
Automating Probabilistic Seismic Hazard Analysis
Calculations, Workflows for e-Science, Springer
2007, XXII, 530 p., 181 illus., Hardcover ISBN
978-1-84628-519-6
107
Pegasus Workflow Mapping
4
1
Original workflow 15 compute nodes devoid of
resource assignment
8
5
9
10
12
13
15
108
Distribution of seismogram jobs
70 hours
109
(No Transcript)
110
Scalability
SCEC workflows run each week using Pegasus and
DAGMan on the TeraGrid and USC resources.
Cumulatively, the workflows consisted of over
half a million tasks and used over 2.5 CPU Years.
Managing Large-Scale Workflow Execution from
Resource Provisioning to Provenance tracking The
CyberShake Example, Ewa Deelman, Scott Callaghan,
Edward Field, Hunter Francoeur, Robert Graves,
Nitin Gupta, Vipin Gupta, Thomas H. Jordan, Carl
Kesselman, Philip Maechling, John Mehringer,
Gaurang Mehta, David Okaya, Karan Vahi, Li Zhao,
e-Science 2006, Amsterdam, December 4-6, 2006,
best paper award
111
CyberShake Computational Results for First 2 sites
  • First two sites (Pasadena and USC) we used NCSA
    SDSC IA-64 machines using
  • 23 days total runtime
  • Condor Glide-in processing approach submitted via
    Pegasus.
  • Failed job recovery
  • Retries
  • Rescue DAG

112
Recent Performance on CyberShake
113
Specific Techniques that Pegasus Uses to Support
Our CyberShake Workflows
  • Workflows have Too Many Jobs
  • Separate in Smaller workflows (Partition) and run
    them one after another
  • Workflows have too many small jobs
  • Bundle jobs together
  • Scheduling of jobs on cluster
  • Schedule Condor Glide-ins on Cluster nodes then
    bypass the PBS scheduler

114
Large, long running, workflows present Planning
Issues.
  • As workflows get large, the decision of when to
    convert to a concrete workflow become quite
    important because conditions change during
    execution of the workflow
  • Advanced (planning) Decide prior to task being
    ready
  • Deferred Decide only when task is ready
  • Mixed mode
  • When you separate abstract (logical) workflow
    from concrete workflow, you (or job planner e.g.
    Pegasus) can decide where to run the jobs at
    run-time.

115
Outstanding Middleware Implications for 3D Wave
Propagation Simulations
  • Schedule reservations on clusters
  • Reservations and special queues are often
    arranged.
  • Large file and data movement
  • TeraByte transfers require high reliably, long
    term, data transfers
  • Ability to stop and restart
  • Can we move restart from one system to another
  • Draining of temporary storage during runs
  • Storage required for full often exceeds
    capability of scratch, so output files must be
    moved during simulation

116
Major Middleware related issues for
SCEC/CMESupercomputing and Storage
  • Globally (TeraGrid wide) visible disk storage
  • Well supported, reliable file transfers with
    monitoring and restart of jobs with problems are
    essential.
  • Interoperability between grid tools and data
    management tools such as SRB must include data
    and metadata and metadata search.

117
Major Middleware related issues for
SCEC/CMEUsability Related and Monitoring
  • Monitoring tools that include status of available
    storage resources.
  • On-the-fly visualizations for run-time validation
    of results
  • Interfaces to workflow systems are complex,
    developer oriented interfaces. Easier to user
    interfaces needed

118
Outline
  • SCEC Earthquake System Science
  • Earthquake System Science Computing
  • Heterogeneous Workflow Research
  • SCEC View of the Computing World

119
  • Simulation program has produced physics-based
    seismic hazard research results that advanced
    SCECs earthquake system science goals.

120
Milestones in Numerical Simulation of Global
Geosystems
  • First long-term climate simulations 1965
  • Coupled ocean-atmosphere climate model 1969
  • First simulations of mantle convection 1974
  • Climate models with ocean eddy-scale
    resolution 1988
  • Realistic model of self-sustaining core
    dynamo 1995
  • Realistic model of mantle convection 1996
  • Fully coupled climate model with interactive
    carbon cycle 2004

Japanese Earth Simulator dedicated March, 2002
121
The Geoscientific Community Is Pursuing a
Petascale Collaboratory
122
SCEC Needs High Performance Computing
  • SCECs science program requires high performance
    computing.
  • We are positioning ourselves as a scientific
    group that can perform important science on
    current and future NSF high performance
    computers.
  • We must show that we are ready to use the next
    generation computers.
  • The next generation of computers are a likely to
    be a huge challenge to acquire, operate, and use.

123
SCEC NSF Supercomputer Allocations
124
Real Crisis With HPC Is With The Software
  • Programming is stuck
  • Arguably hasnt changed since the 60s
  • Its time for a change
  • Complexity is rising dramatically
  • highly parallel and distributed systems
  • From 10 to 100 to 1000 to 10000 to 100000 of
    processors!!
  • multidisciplinary applications
  • A supercomputer application and software are
    usually much more long-lived than a hardware
  • Hardware life typically five years at most.
  • Fortran and C are the main programming models
  • Software can be a major cost component of modern
    technologies.
  • The tradition in HPC system procurement is to
    assume that the software is free.
  • We dont have many great ideas about how to solve
    this problem.

125
SCEC Pursuing Leadership Class Computer Systems
Data-oriented Science and Engineering
Environment
100 TF Systems 10s of Projects
Key function of the NSF Supercomputer Centers
Provide facilities over and above what can
be found in the typical campus/lab environment
10s of 10 TF Systems 1,000s of Users
HPC Centers
Data (more BYTES)
100s of 1 TF Systems 10,000s of Users
Home, Lab, Campus, Desktop
TraditionalHPC environment
Departmental HPC
GigaFLOPS Millions of Users
Workstations
Compute (more FLOPS)
126
AWM Olsen (2006) 40960 Processors
  • Regional M8.1 domain
  • 32 billion mesh points
  • outer/inner scale ratio of 8,000
  • 40,960 BGW processors
  • 96 efficiency
  • 6.1 Tflops

Yifeng Cui, SDSC F06 AGU
40,960 processors
127
Along with scaling to large processor count, we
are also improving codes that can express
geometric complexity
128
Collection of Codes used to run AWM-Olsen Finite
Difference 3D Wave Propagation Software
129
SCEC Computational Platform Concept
  • Our experience is that in order to get a research
    result, it requires more than a single program.
  • Input data
  • Multiple simulation codes
  • Verification and validation work
  • Knowledgable users
  • Software infrastructure
  • Out of this experience, the concept of a
    computational platform has emerged to refer to
    a collection of codes that can be used to produce
    a specific research result.

130
SCEC Computational Platform Concept
  • Definition of Computational Platform
  • A vertically integrated collection of hardware,
    software, and people that provides a broadly
    useful research capability
  • Implied capabilities
  • Validated simulation software and geophysical
    models
  • Re-usable simulation capabilities
  • Imports parameters from other systems. Exports
    results to other systems
  • IT/geoscience collaboration involved in operation
  • Access to High-performance hardware and large
    scale data and metadata management.
  • May use Workflow management tools

131
SCEC Computational Platforms
132
SCEC Simulation Projects begin with Scientific
Objectives
  • Extend deterministic simulations of strong ground
    motions to 3 Hz for investigating the upper
    frequency limit of deterministic ground-motion
    prediction.
  • Improve the resolution of dynamic rupture
    simulations by an order of magnitude for
    investigating the effects of realistic friction
    laws, geologic heterogeneity, and near-fault
    stress states on seismic radiation.
  • Compute physics-based PSHA maps and validate
    those using seismic and paleo-seismic data.

133
PetaSHA Milestone Simulations
Simulation Volumes V1 Northridge
domain V2 PSHA site volume V3 regional M7.7
domain V4 regional M8.1 domain
http//scecdata.usc.edu/petasha
134
(No Transcript)
135
(No Transcript)
136
Petascale computing will be needed for SHA
simulations
137
Possible Science and Cyberinfrastructure Lifecycle
New Science Researchgoals
motivate
enable
New infrastructurecapabilities
New infrastructurecapabilities
New Science Researchgoals
enable
motivate
138
Possibly Improved Science and Cyberinfrastructure
Lifecycle Advancement of Knowledge
New Science Researchgoals
New Knowledge and Understanding
New Science Researchgoals
enable
motivate
enable
motivate
New infrastructurecapabilities
New infrastructurecapabilities
139
End
140
Abstract Workflow Reduction
141
Optimizing from the point of view of Virtual Data
Job c
Job a
Job b
Job f
Job e
Job d
Job g
Job h
Job i
  • Jobs d, e, f have output files that have been
    found in the Replica Location Service.
  • Additional jobs are deleted.
  • All jobs (a, b, c, d, e, f) are removed from the
    DAG.

142
Planner picks execution and replica
locations Plans for staging data in
Job c
adding transfer nodes for the input files for the
root nodes
Job a
Job b
Job f
Job e
Job d
Job g
Job h
Job i
143
Staging data out and registering new derived
products in the RLS
Job c
Job a
Job b
Job f
Job e
Job d
Job g
Job h
Staging and registering for each job that
materializes data (g, h, i ).
Job i
KEY The original node Input transfer
node Registration node Output transfer
node Node deleted by Reduction algorithm
144
(No Transcript)
Write a Comment
User Comments (0)
About PowerShow.com