Report from Amsterdam

1
Report from Amsterdam

• Massimo Sgaravatto
• INFN Padova

2
Amsterdam events

• Global Grid Forum (GGF) 1
• DataGrid workshop

3
Global Grid Forum

• Focuses on the promotion and development of Grid
  technologies and applications via the development
  and documentation of "best practices,"
  implementation guidelines, and standards
• Global GF efforts are also aimed at the
  development of a broadly based Integrated Grid
  Architecture that can serve to guide the
  research, development, and deployment activities
  of the emerging Grid communities
• www.globalgridforum.org

4
Global Grid Forum History
1999
2000
GF BOF (Orlando)
GF1 (San Jose NASA Ames)
GF2 (Chicago iCAIR)
eGrid and GF BOFs (Portland)
GF3 (San Diego SDSC)
eGrid1(Posnan PSNC)
GF4 (Redmond Microsoft)
Asia-Pacific GF Planning (Yokohama)
eGrid2 (Munich at Europar)
GF5 (Boston Sun)
Global GF BOF (Dallas)
5
Global Grid Forum 2001
2001
GGF1 (Amsterdam) March 4-7, 2001 w/ DataGrid
Kickoff (March 7-9) Hosts Amsterdam Science
and Technology Center (SARA, Univ. Amsterdam,
NIKHEF) 325 participants 192 organizations 28
countries
GGF2 (Washington, DC) July 15-18, 2001
GGF3 (Rome) October 7-10, 2001 W/ DataGrid
workshop (October 2-5) Frascati (Rome)
6
Global GF Working Groups

• Account Management
• Applications and Testbeds
• Grid Information Services
• Remote Data Access
• Security
• Advanced Collaborative Environments
• Grid Protocol Architecture

• Advanced Programming Models
• Grid Computing Environments
• Grid Performance
• Scheduling
• User services
• JINI

7
How to Be Involved

• Electronic Participation
• Each working group uses email distribution lists
  for discussions
• Contributing to Grid Working Drafts
• Volunteer to help write a planned paper
• Submit an abstract/outline to propose a paper
• Architectures / Models
• API, Protocol, or other specification
• Survey or Current Practice Description
• Lead a Group
• Volunteer to start a working group or a sub-group
  within an existing working group

8
Scheduling Resource Management WG

• Charter generate best practice scheduling and
  resource management documents and API
  specifications to enable interoperability
• Co-chairs Jenny Schops, Bill Nitzberg, Uwe
  Schwiegelshohn
• Web site http//www.cs.nwu.edu/jms/sched-wg
• Mailing list sched-wg_at_gridforum.org
• General meetings at Grid forum events and
  teleconferences

9
Scheduling WG _at_ GGF 1

• 3 documents discussed
• 10 actions for superscheduling
• Goal list the basic steps and capabilities
  involved in resource reservation, acquisition and
  use
• Basically done
• Advanced reservation API
• Just an API because not yet advanced for a
  protocol
• Basically done
• Attributes for communication about scheduling
  instances
• Goal define a set of attributes that a higher
  level scheduler can ask of a lower level
  scheduler
• In progress

10
Scheduling WG _at_ GGF 1

• Suggested new documents
• Advance reservation protocol
• Allow advance reservation between systems
• Scheduling dictionary
• List of tokens and definitions to coordinate with
  the dictionary group (common glossary of
  important terms across the Grid forum)
• Already started (terms only, no definitions)
  (www.sdsc.edu/mthomas/GF/terms/sched-terms.cgi)
• Document to move this effort forward

11
Distributed Accounting WG

• Purpose
• Develop a prototype Grid accounting architecture
  such as is required for resource economies
• Define interactions with other Grid components
  and existing systems
• Create a specification that can be used to build
  and evaluate prototype systems
• Co-chairs Tom Hacker, William Thigpen
• Web site http//www.nas.nasa.gov/thigpen/account
  s-wg
• Mailing list accounts-wg_at_gridforum.org
• Working drafts we may be interested in
• Distributed accounting on the Grid
• An economy Grid architecture for service-oriented
  grid computing

12
GGF and WP 1

• Should we/how do we contribute to the GGF effort
  ?
• Dictionary ?
• Accounting ?
• ?

13
DataGrid

• ATF Progress Report
• Joint Parallel meetings
• WP 1-5 (Middleware)
• WP 6-7 (Testbed)
• WP 8-10 (Applications)

14
ATF Progress Report

• Documents (http//gridatf.web.cern.ch/grid-atf/doc
  uments.html)
• DataGrid Architecture version 1
• Month 9 release
• We have to provide reactions
• Implementation plan for M9 release by the next
  PTB (April 19)

15
WP 1 5

• Information Services
• Relational vs. LDAP
• Scope
• WP 3 is going to provide only APIs (for producers
  and consumers) and interfaces
• What will be there in M9 ?
• Current Globus GIS implementation (WP 3)
• WP 1 many shortcomings and open issues
  (performances, incomplete schema,) in the
  current Globus GIS implementation, and therefore
  it is probably unsuitable
• Discussion with S. Tuecke (Globus team)
• Aware of problems
• We may use/test Globus alpha releases
• ? Workshop (Convenior WP 3 )
• April 28,29 _at_ Cern

16
WP 1 - 5

• Job
• Partition/Checkpoint of jobs ?
• At GRID level but also inside fabric ?
• ? Applications have to help us, otherwise it is
  difficult how to split/checkpoint jobs
• Job input/output
• Files ? Objects ? ?
• Is the input/output known at schedule time ?
• Scheduling policies ?
• Deadline? Best fit ?, ?
• ? Applications should provide their view

17
WP 1 5

• Storage Model
• Boundaries between WP2, WP4, WP 5
• Global vs. local filenames
• Who does the house keeping
• ? Workshop Convenior WP 2
• March 27, 2001 _at_ Cern ?
• Accounting
• Economic model (WP 1) ?
• ? WP 1 to investigate and circulate thoughts

18
WP 6 7

• Testbed organization
• Permanent Group (partner representatives,
  technical committee, site contacts, testbed
  support)
• Release team (one team for each release
  validation phase)
• Working model
• WP test activity (WPx)
• WP release (WPx testbed appl.)
• Testbed release (WPs testbed appl.)
• Official testbed release ( users)
• Waiting for test requests

19
WP 6 7

• Testbed 0
• Grid basic services deployment, based on Globus
  toolkit
• Resources
• Mainly Linux machines (also Suns, SGI)
• LSF, Condor, PBS, BQS
• gt 100 people
• gt 40 sites
• Main issues
• Information Services
• Resource access (certification, security and
  policies management)
• Installation toolkit

20
GIS Topology
(root server at Cern)

EU GIIS odatagrid0
(Top level server)
LIP (Portugal) dclip,dcpt,odatagrid0
INFN (Italy) dcinfn,dcit,odatagrid0
France dcfr,odatagrid0
...
...
...
...
...
dcin2p3,dcfr,odatagrid

• Support for GIS deployment in place

21
WP 6 7

• Globus installation toolkit
• 3 toolkits available (INFN, UK, FR)
• ? Working group to converge on common tools and
  common environment (within 3 weeks)
• Security
• CA group (chaired by D. Kelsey) already in place
• Support for certificates signed by the various
  national CAs
• Access policies for Testbed 0
• Globus grid-mapfiles
• Only one account per certificate (no group
  accounts)
• Requests for accounts for Testbed 0 via WP
  managers to WP6
• General security group (chaired by D. Kelsey) is
  going to set up
• Security Workshop (29, 30 March _at_ Cern,
  Convenior D. Kelsey)
• Representatives from WPs
• Security experts/officers/reps from Testbed sites
• Other interested people

22
WP 8 - 10
Specific application layer
ALICE
ATLAS
CMS
LHCb
WP9
WP 10
VO common application layer
WP8-9-10 TWG
HEP
Earth Obs.
Biology
DataGRID middlewarePPDG, GriPhyn, EuroGRID
Bag of Services (GLOBUS)
OS Net services
23
WP 8 10

• User cases for testbed 0 1
• Basically ready for WP 8
• Available for WP 9
• User cases for testbed 2 3
• Draft in one month for WP 8
• Common (WP8-10) user requirements
• First draft in one month
• First release at M6

24
WP 8 10

• Preliminary reactions to ATF
• Objects or files when interacting with the
  middleware ?
• Access to middleware services from applications
  via API (C API)
• RW databases
• Interactive jobs
• They will propose use cases
• Required job resources unknown at schedule time
• Middleware portability
• Linux and Solaris is not enough
• The code should be written to run on any machine
  with a reasonable cc/java
• MPI and Corba
• Worries about the use of Condor (open source)

25
Other issues (not tackled)

• Grid global queues or local queues
• Gateway functionality
• Sharing of WP1 and WP4 tasks on the gateway
• Language translation
• Translation of Grid credentials to local
  credentials
• WP4 Will other WPs only query for information of
  the current state or is there a need to query for
  future changes ?
• Central middleware repository
• Coding convention and tools

26
Informal discussions

• WP 2
• Agent-based query optimization
• WP 4
• For month 9 they basically agree to consider
  queues as resources
• Resource monitoring info
• We are consumer, they are producer
• We have to specify the attributes needed for M 9
  release, and they will tell us if they can
  implement the corresponding information providers
• depends on what exactly WP 3 plans to deliver
  concerning information and monitoring service

27
Condor

• In the first year we are allowed to use the
  Condor stuff
• Then we can rely only on open source software
• Probably possible for the Condor matchmaking
  library
• More complicated for Condor-G
• Possibility to have the source code

28
Condor whats new

• New Condor grid-manager and old Globus job
  manager (basically ready)
• Fixes for existing shortcomings (scalability in
  the submitting machine, logging, )
• Globus GRAM client APIs
• New Globus job manager (in testing phase)
• 2-phase submit protocol
• Support for reattaching to an existing job
• New Condor grid-manager (4-6 weeks)
• Able to exploits the new Globus job manager

29
Agenda

• DataGrid topical (IS, DM, Security) workshops at
  CERN (week of March 26)
• DataGrid technical visit to Globus at ANL April
  5-7
• Agent technology workshop in Trento (ITC/IRST)
  May 17
• INET 2001 5-8 June in Stockholm
• EuroGlobus 16-22 June
• EU-US coordination in Rome on 23/6/2001
• Next project conference at Oxford 2-5 of July
• GGF2 in Washington 15-18 of July
• HPDC 7-10 of August in the US

30
Agenda

• CHEP2001 in Beijing (Grid sessions) 3-7 of
  September
• CERN school of Computing (Grid session) 16-29
  September in Santander, Spain
• 6th Globus retreat and iDG-1 17-21 of September
  in Chicago
• DataGridGGF3 in FrascatiRome 2-5 7-10 of
  October
• IST2001 in Brussels end of October-early November
  (DataGrid booth, presentations and demos?)
• SC2001 10-16 November (DataGrid participation?)