dev.globus a framework for open Globus development

1
dev.globusa framework for open Globus
development
Dr. Jennifer M. Schopf UK National eScience
Centre Argonne National Laboratory
2
Globus Open Source History

• 1996 First established as an open source
  software project
• Globus development team limited to ANL/UC and ISI
• 2003 Globus Alliance formed
• EPCC (Edinburgh, UK), NCSA (IL, USA), and KTH
  (Sweden)
• 2005 dev.globus.org formed
• source code development infrastructure
• meritocratic governance model

3
Outline

• What is Globus
• A little scene setting
• Some technology highlights
• The dev.globus environment
• A little on the governance and infrastructure
• Kinds of projects
• Incubator projects

4
What is a Grid?

• Resource sharing
• Computers, storage, sensors, networks,
• Sharing always conditional issues of trust,
  policy, negotiation, payment,
• Coordinated problem solving
• Beyond client-server distributed data analysis,
  computation, collaboration,
• Dynamic, multi-institutional virtual orgs
• Community overlays on classic org structures
• Large or small, static or dynamic
• Hard because of lack of central control, shared
  resources, crossing of administrative domains

5
The Role of the Globus Toolkit

• A collection of solutions to problems that come
  up frequently when building collaborative
  distributed applications
• Heterogeneity
• A focus, in particular, on overcoming
  heterogeneity for application developers
• Standards
• We capitalize on and encourage use of existing
  standards (IETF, W3C, OASIS, GGF)
• GT also includes reference implementations of
  new/proposed standards in these organizations

6
HeterogenietyGlobus is an Hour Glass
Higher-Level Services and Users

• Local sites have an their own policies, installs
  heterogeneity!
• Queuing systems, monitors, network protocols, etc
• Globus unifies
• Build on Web services
• Use WS-RF, WS-Notification to represent/access
  state
• Common management abstractions interfaces

Standard GT4 Interfaces
Local heterogeneity
7
StandardsWhat does WSRF give Globus

• Reference implementation of WSRF and WS-N
  functions
• Naming and bindings (basis for virtualization)
• Every resource can be uniquely referenced and has
  one or more associated services for interacting
• Lifecycle (basis for resilient state management)
• Resources created by svcs following a factory
  pattern
• Resource destroyed immediately or scheduled
• Information model (basis for monitoring
  discovery)
• Resource properties associated with resources
• Operations for querying and setting this info
• Asynchronous notification of changes to
  properties
• Service groups (basis for registries collective
  svcs)
• Group membership rules and membership management
• Base fault type

8
Standards WSRF vs XML/SOAP

• The definition of WSRF means that the Grid and
  Web services communities can move forward on a
  common base
• Why Not Just Use XML/SOAP?
• WSRF and WS-N are just XML and SOAP
• WSRF and WS-N are just Web services
• Benefits of following the specs
• These patterns represent best practices that have
  been learned in many Grid applications
• There is a community behind them
• Why reinvent the wheel?
• Standards facilitate interoperability

9
Globus is a Building Block

• Basic components for grid functionality
• Highest-level services are often application
  specific, we let applications concentrate there
• Easier to reuse than to reinvent
• Compatibility with other Grid systems comes for
  free
• We provide basic infrastructure to get you one
  step closer

10
Globus Toolkit Open Source Grid Infrastructure
XIO
Core WS schema
Globus Projects dev.globus.org
Data Replication
Java WS Core
Replica Location
Virtual Workspaces
Delegation
COG jGlobus
Incubator Management
Reliable File Transfer
CAS/SAML Utilities
C WS Core
Dynamic Accounts
GT Distribution
GridFTP
C Security
Grid Resource Allocation Management
MDS4
C Core Utilities
GT Release Manuals
Data Mgmt
Security
CommonRuntime
Execution Mgmt
Info Services
Other Projects
11
Data MgmtGridFTP
Disk-to-disk onTeraGrid

• A high-performance,
• secure, reliable trans.
• protocol optimized for
• high-bandwidth WANs
• IPv6 Support
• XIO for different transports
• Striping ? multi-Gb/sec wide area transport
• Pluggable
• Front-end e.g., future WS control channel
• Back-end e.g., HPSS, cluster file systems
• Transfer e.g., UDP, NetBLT transport

12
Data Mgmt Reliable File Transfer

• A WSRF service for queuing file transfer requests
• Server-to-server transfers
• Checkpointing for restarts
• Database back-end for failovers
• Integrated failure recovery
• Allows clients to requests transfers and then
  disappear
• No need to manage the transfer
• Status monitoring available if desired

13
Data ManagementReplica Location Service

• Identify location of files via logical to
  physical name map
• Distributed indexing of names, fault tolerant
  update protocols
• GT4 version scalable stable
• Managing 40 million files across 10 sites

Index
Index
14
Execution Management GRAM

• Common WS interface to schedulers
• Unix, Condor, LSF, PBS, SGE,
• More generally interface for process execution
  management
• Lay down execution environment
• Stage data
• Monitor manage lifecycle
• Kill it, clean up
• Not a scheduler/broker, but an uniform,
  web-service interface to local resources

15
Information Services MDS4

• Monitoring and Discovery System
• Set of information providers
• Interfaces to data sources any web service,
  cluster systems, queuing systems, archives, file
  scraping, etc
• Higher level tools
• Index Service registry with cache
• Trigger Service warning system
• GUI interface - WebMDS

16
(No Transcript)
17
Globus Toolkit Open Source Grid Infrastructure
XIO
Core WS schema
Globus Projects dev.globus.org
Data Replication
Java WS Core
Replica Location
Virtual Workspaces
Delegation
COG jGlobus
Incubator Management
Reliable File Transfer
CAS/SAML Utilities
C WS Core
Dynamic Accounts
GT Distribution
GridFTP
C Security
Grid Resource Allocation Management
MDS4
C Core Utilities
GT Release Manuals
Data Mgmt
Security
CommonRuntime
Execution Mgmt
Info Services
Other Projects
18
Outline

• What is Globus
• Some technology highlights
• The dev.globus environment
• A little on the governance and infrastructure
• Kinds of projects
• Incubator projects

19
Why is Globus Software Open Source?

• To allow for inspection
• For consideration in standardization processes
• To encourage adoption
• In pursuit of ubiquity and interoperability
• To encourage contributions
• Harness the expertise of the community

20
Open Contribution

• But distributing code under an open source
  license does not guarantee open development!
• Open development requires open processes
• So we have created dev.globus to facilitate
  contributions
• http//dev.globus.org/

21
Governance Model

• Based on Apache Jakarta
• Individual development efforts organized as
  projects
• Consensus-based decision making
• Control over each project in the hands of its
  most active and respected contributors
  (committers)
• Globus Management Committee (GMC) providing
  overall guidance and conflict resolution

22
Common Infrastructure

• Code repositories (CVS, SVN)
• Mailing lists
• -dev, -user, -announce, -commit for every
  project
• Issue tracking (bugzilla)
• Including roadmap info for future development
• Wikis
• Known interactions for people accessing your
  project

23
Sample

• http//dev.globus.org/wiki/GRAM

24
(No Transcript)
25
(No Transcript)
26
(No Transcript)
27
(No Transcript)
28
Technology Projects

• Common runtime projects
• C Core Utilities, C WS Core, CoG jglobus, Core WS
  Schema, Java WS Core, XIO
• Data projects
• GridFTP, Reliable File Transfer, Replica
  Location, Data Replication
• Execution projects
• GRAM, Dynamic accounts, Virtual workspaces
• Information services projects
• MDS4
• Security Projects
• C Security, CAS/SAML Utilities, Delegation
  Service, MyProxy

29
Non-Technolgy Projects

• Distribution Projects
• Globus Toolkit Distribution
• Process was used for April 4.0.2 release
• Documentation Projects
• GT Release Manuals
• Incubation Projects
• Incubation management project
• And any new projects wanting to join

30
Incubator Process in dev.globus

• Entry point for new Globus projects
• Incubator Management Project (IMP)
• Oversees incubator process form first contact to
  becoming a Globus project
• Quarterly reviews of current projects
• Process being debugged by Incubator Pioneers
• http//dev.globus.org/wiki/IncubatorDraft

31
Process (1 of 3)

• Project proposes itself as a Candidate
• A proposed name for the project
• A proposed project chair, with contact info
• A list of the proposed committers for the
  project
• An overview of the aims of the project
• An overview of any current user base or user
  community, if applicable
• An overview of how the project relates to other
  parts of Globus
• A summary of why the project would enhance and
  benefit Globus.

32
Process (2 of 3)

• IMP meet, discuss, and accept project as a
  ProtoProject
• ProtoProject now part of the Incubator framework
• Get assigned a Mentor to help
• Member of IMP
• Bridge between Globus and new ProtoProject
• Opportunity to get up to speed on Globus
  Development process

33
Process (3 of 3)

• Quarterly reviews by IMP determine
• Stay a ProtoProject
• Retire
• Escalate to a full Globus project
• Escalation when ProtoProject passes checklist
• Legal
• Meritocracy
• Alignment/Synergy
• Infrastructure

34
Current List of Incubator Pioneers

• Metrics
• Project for usage stats and usage metrics for GT4
• Lee Liming, University Chicago
• GridWay
• MetaScheduler that works over GT2 and GT4 GRAM,
  MDS
• Ignacio Lorente, University Madrid
• GridShib
• Integration of GSI and Shibboleth security
• Von Welch, NCSA
• Send email to incubator-committers_at_globus.org
  to discuss new project ideas!

35
You Can Begin Participating in Globus
Development Today!

• Monitor and comment on Globus development
  discussions recent threads include
• GT Backward Compatibility (gt-dev_at_globus.org)
• 4.2 Wish List for GRAM (gram-dev_at_globus.org)
• Submit bug fixes and other contributions
• Start your own Globus project!

36
Globus Toolkit Open Source Grid Infrastructure
XIO
Core WS schema
Globus Projects dev.globus.org
Data Replication
Java WS Core
Replica Location
Virtual Workspaces
Delegation
COG jGlobus
Incubator Management
Reliable File Transfer
CAS/SAML Utilities
C WS Core
Dynamic Accounts
GT Distribution
GridFTP
C Security
Grid Resource Allocation Management
MDS4
C Core Utilities
GT Release Manuals
Data Mgmt
Security
CommonRuntime
Execution Mgmt
Info Services
Other Projects
37
Summary

• Globus has been an open source toolkit for 10
  years
• Our recent move to dev.globus.org will make it
  much easier for outside contributors and projects
• Please consider helping out!

38
Credits

• Our guidelines are derived directly from those of
  the Jakarta project and the Apache project within
  which Jakarta lives (with modifications to
  reflect specific properties of the Globus
  community)
• Thanks to Jakarta and Apache for the effort they
  have put into developing and documenting their
  processes!

39
More Information

• Jennifer M. Schopf
• jms_at_mcs.anl.gov
• Globus Development Environment
• dev.globus.org
• Globus WORLD 2006
• http//www.globusworld.org/
• Washington D.C. Convention Center
• September 11 - 15, 2006
• Tenth Birthday Party youre invited!