Scientific Workflow management in Virtual Laboratory in e-Science (VL-e)

1
Scientific Workflow management in Virtual
Laboratory in e-Science (VL-e)

• Dr Zhiming Zhao (zhiming_at_science.uva.nl)
• Prof. Dr. Bob Hotzberger (bob_at_science.uva.nl)
• System and Network Engineering, University of
  Amsterdam
• Virtual Laboratory for e-Science

20 Intl Conf. CODATA 2006, Beijing, China
2
Outline

• Background
• Scientific workflow management systems
• Virtual Laboratory for e-Science
• Our approach
• Challenges and research lines
• Activities

3
Scientific experiments and support systems
4
Scientific workflow systems a new GUISE of
Problem Solving Environments

• In our view, a SWMS at least implements
• A model for describing workflows
• An engine for executing/managing workflows
• Different levels of support for a user to
  compose, execute and control a workflow.

Workflow (based on certain model)
Composition
A SWMS
User support
Engine level control
Engine
Resource level control
resources
5
Diversity in SWMSs

• Taverna
• Web services based language Scufl
• FreeFluo engine
• Graphical viz of workflow

• Triana
• Components
• Task graph
• Data/control flow

• Kepler
• Actor, director
• MoML
• Execution models

• Pegasus
• Based on DAGMan
• VDL
• DAG

• VLAMG
• Moudules
• Data streams

• DAGMan
• Computing tasks
• DAG

6
Research mission

• Effectively reuse existing workflow management
  systems, and provide a generic e-Science
  framework for different application domains.
• A generic framework can
• Improve the reuse of workflow components and the
  workflows for different experiments
• Reduce the learning cost for different systems
• Allow application users to work on a consistent
  environment when underlying infrastructure is
  changed
• Promote knowledge transfer between scientists and
  between domains

7
Research contextVirtual Laboratory for
e-Science (VL-e)(input from Bob.)
Data intensivescience
Bioinformatics ASP

• Challenges in VL-e
• Between rapid prototype and proof of concept
• between infrastructure and application
• Between academic and industry

Application layer
Generic e-science framework layer
Grid layer
8
Previous work VLAM-G PFT/Study

• Process-Flow Template
• Graphical representation of data elements and
  processing steps in an experimental procedure.

• Study
• Descriptions of experimental steps represented
  as an instance of a PFT with references to
  experiment topologies.

• Experiment Topology
• Graphical representation of self-contained data
  processing modules attached to each other in a
  workflow.

9
Lessons learned

• How to introduce a new PSE to a domain scientist?
• Because it has a beautiful architecture?
• Or because it can allow a scientist to keep their
  current work style?
• How to use existing work?
• Scientists need one system or more options?
• How to include user in the computing loop?
• Dynamic workflows and human in the loop computing
  are important.

Z. Zhao et al., Scientific workflow management
between generality and applicability, QSIC 2005,
Australia
10
Workflow support in VL-e

• Recommend suitable workflow systems for different
  application domains
• Analyze typical application use cases
• Define small projects with different application
  domains
• Review existing workflow systems
• Recommend four workflow systems Triana, Taverna,
  Kepler, and VLAMG
• A long term
• Extend VLAMG and develop our own generic workflow
  framework
• A workflow bus architecture is proposed.

Z. Zhao et al, VLWF-Bus a workflow bus for
e-Science applications the 2nd IEEE e-Science
and Grid computing, Amsterdam, 2006.
11
Use case story (to be added)
12
Discussion

• Challenges in supporting scientific workflows
• Requirements on domain specific experiments
• Generic workflow support and domain specific
  applications
• Existing workflow management systems are diverse
  in functionality, design and user support
• Related work
• Interoperability among workflow systems (sister
  Link project)
• Resource level e.g., Kepler invokes Tavernas
  resources

13
Summary
14
Activities

• Intl workshop on Workflow systems in
  e-Science, organized by Zhiming Zhao and Adam
  Belloum, in the context of ICCS, 2006 Reading
  University, 2007 Beijing, China.
• Proceedings is in LNCS, Springer Verlag.
• A special issue will be published in Scientific
  Programming Journal.
• http//staff.science.uva.nl/zhiming/iccs-wses
• Workshop on Scientific workflows and industrial
  workflow standards in e-Science , organized by
  Adam Belloum and Zhiming Zhao, in the context of
  IEEE e-Science and Grid computing conference in
  Amsterdam December 2006.
• Pegasus, Dr. Ewa Deelman (Department of Computer
  Science University of South California)
• BPEL, Dr. Dieter König (IBM Research Germany
  Development Laboratory)
• Kepler, Dr. Bertram Ludäscher (Department of
  Computer Science University of California, Davis)
  
• Taverna, Prof. Peter Rice (European
  Bioinformatics Institute)
• WS and Semantic issues, Dr. Steve Ross-Talbot
  (CEO, and a co-founder, of Pi4 Technologies)
• Triana, Dr. Ian J. Taylor (Department of Computer
  Science Cardiff University)
• http//staff.science.uva.nl/adam/workshop/VL-e-wo
  rkshop.htm

15

• References
• Z. Zhao A. Belloum H. Yakali P.M.A. Sloot and
  L.O. Hertzberger Dynamic Workflow in a Grid
  Enabled Problem Solving Environment, in
  Proceedings of the 5th International Conference
  on Computer and Information Technology (CIT2005),
  pp. 339-345 . IEEE Computer Society Press,
  Shanghai, China, September 2005.
• Z. Zhao A. Belloum A. Wibisono F. Terpstra
  P.T. de Boer P.M.A. Sloot and L.O. Hertzberger
  Scientific workflow management between
  generality and applicability, in Proceedings of
  the International Workshop on Grid and
  Peer-to-Peer based Workflows in conjunction with
  the 5th International Conference on Quality
  Software, pp. 357-364. IEEE Computer Society
  Press, Melbourne, Australia , September 19th-21st
  2005.
• Z. Zhao A. Belloum P.M.A. Sloot and L.O.
  Hertzberger Agent Technology and Generic
  Workflow Management in an e-Science Environment,
  in Hai Zhuge and G.C. Fox, editors, Grid and
  Cooperative Computing - GCC 2005 4th
  International Conference, Beijing, China, in
  series Lecture Notes in Computer Science, vol.
  3795, pp. 480-485. Springer, November 2005. ISBN
  3-540-30510-6. (DOI 10.1007/11590354_61)
• Z. Zhao A. Belloum P.M.A. Sloot and L.O.
  Hertzberger Agent technology and scientific
  workflow management in an e-Science environment,
  in Proceedings of the 17th IEEE International
  conference on Tools with Artificial Intelligence
  (ICTAI05), pp. 19-23. IEEE Computer Society
  Press, Hongkong, China, November 14th-16th 2005.
• Z. Zhao Suresh Booms A. Belloum P.M.A. Sloot
  and L.O. Hertzberger VLWF-Bus a workflow bus
  for e-Science applications, in Proceedings of the
  2nd IEEE e-Science and Grid computing, IEEE
  Computer Amsterdam, December 46 2006.