Workflow Management

1
Workflow Management

• Concepts and Requirements
• For Scientific Applications

2
The Two Tier View of a Workflow Management system
(generic)
3
Workflow layers

• Control-flow layer
• Application and Software Tools layer
• I/O System layer
• Storage and Network Resource layer

4
Anatomy of a scientific workflow management
system
Task A Generate Time-Steps
Flow Tier
Task B Move TS
Task D Visualize TS
Task C Analyze TS
Control Flow Layer

Applications Software Tools Layer
Data Mover
Parallel R
Post Processing
Terascale Browser
Simulation Program
Work Tier
I/O System Layer
HDF5 Libraries
Parallel NetCDF
PVFS
LN
SRM
Storage Network Resouces Layer
5
Architecture of a workflow management system
Post run time
Design time (Process modeling)
Run time
Workflow engine
Tracking Tools
Data Mining Analysis Tool
Process Design Tool
Workflow executionhistory database
Process Database
Invoke Tasks
Application code Software Tools
6
Main Services Offered by Workflow Management
Systems

• Workflow design tools (GUI interfaces)
• Automatic sequencing of component invocation
• Synchronizing data flow between components
• Tracking and reporting mechanisms

7
Discussion items

• Control flow tier
• Granularity of tasks, sub-workflows
• Task Invocation - Web Services, Corba, Wrappers,
  Callbacks
• Human tasks Notifications and alerts, steering
• Dataflow streaming granularity
• Performance expectations (provoke alerts)
• Work Tier
• Workflow engine for scientific applications
• Integrated dataflow management
• Failure detection and recovery
• Data-driven flow control
• Performance-driven flow control
• Workflow optimization
• Run-time resource coordination

8
Discussion items
Deployment and maintenance
Research and Development
Hardening and Packaging

• - Granularity of tasks, sub-workflows
• - Task Invocation
• - Human tasks Notifications, alerts, steering
• - Dataflow streaming granularity
• - Performance expectation
• - Workflow engine for scientific applications
• - Integrated dataflow management
• - Failure detection and recovery
• - Data-driven flow control
• - Performance-driven flow control
• - Workflow optimization
• - Run-time resource coordination

X
X
X
X
X
X
X
X
X
X
X
X
9
Other topic discussed

• Simulations setup
• reserve resources, schedule run
• Performance monitoring needed
• e.g. Disks getting full
• e.g. Some task is stalled or too slow
• Interoperation of components
• Match outputs to inputs (e.g. CCA)
• Allow feedback loops
• How to specify what to do in case of
  failures/exceptions

10
Example Workflow Systems Considered

• Scientific
• Ptolemy/Kepler UC Berkeley
• DAGMAN U Wisc, used in Griphyn
• SciRun U Utah, collaborate with CCA
• (Taverna)
• (Triana)
• Commercial
• FileNet
• Oracle Workflow
• IBMs MQ flow