Grid Middleware and prospects on high level of standardization

1
Grid Middleware and prospects on high level of
standardization
and p
e
2
Why is Grid MiddlewareSuch a Challenge?(the
road from CHEP2000 to CHEP2003)
3
Because developing good software is not easy
and Distributed Computingis a hard
problem!(do not try it at home!)
4
Claims for benefits provided by Distributed
Processing Systems
P.H. Enslow, What is a Distributed Data
Processing System? Computer, January 1978

• High Availability and Reliability
• High System Performance
• Ease of Modular and Incremental Growth
• Automatic Load and Resource Sharing
• Good Response to Temporary Overloads
• Easy Expansion in Capacity and/or Function

5
Benefits to Science

• Democratization of Computing you do not have
  to be a SUPER person to do SUPER computing.
• Speculative Science Since the resources are
  there, lets run it and see what we get.
• Data Mining Lets do a pair wise comparison of
  these 120,000 proteins
• Function shipping Find the image in this 3 TB
  collection that has a red car.

6
CERN 92
7
The 94 Worldwide Condor Flock
Amsterdam
Delft
3
30
10
200
3
3
3
Madison
Warsaw
10
10
Geneva
Dubna/Berlin
8
The NUGn Quadratic Assignment Problem (QAP)
aijbp(i)p(j)
min p??
9
Solution Characteristics (n30)
Scientists 4
Wall Clock Time 6220431
Avg. workers 653
Max. workers 1007
CPU Time Approx. 11 years
Nodes 11,892,208,412
LAPs 574,254,156,532
Parallel Efficiency 92
10
CMS Integration Grid Testbed Managed by ONE
Linux box at Fermi
11
Why is this so hard?

• Declarative rather than procedural definition of
  requests.
• Submit and forget.
• Effective planning of request execution.
• Trustworthy execution of requests.
• Distributed ownership of resources.
• Physical distribution of resources.

12
The Ethernet Protocol

• IEEE 802.3 CSMA/CD - A truly distributed (and
  very effective) access control protocol to a
  shared service.
• Client responsible for access control
• Client responsible for error detection
• Client responsible for fairness

13
Client Responsibilities

• Use algorithms that can generate very large
  numbers of independent tasks use pleasantly
  parallel algorithms
• Implement self-contained portable workers this
  code can run anywhere!
• Detect failures and react gracefully use
  exponential back off!
• Be well informed and opportunistic get your
  work done and out of the way!

14
The Layers of Condor
Application
Submit (client)
Application Agent
Customer Agent
Matchmaker
Owner Agent
Execute (service)
Remote Execution Agent
Local Resource Manager
Resource
15
Grid
WWW
Master
Worker
16
Being a Master

• Customer deposits task(s) with the master that
  is responsible for
• Obtaining resources and/or workers
• Deploying and managing workers on obtained
  resources
• Assigning and delivering tasks to
  obtained/deployed workers
• Receiving and processing results

17
Client
Master
MW Library (master)
Condor-G
Communication Library
GT-Gate Keeper
StartD
18
PlanningandExecuting
19
Customer orders agentPlace y F(x) at
L!Grid delivers.
20
A simple plan for yF(x) -gt L

1. Allocate (size(x)size(y)size(F)) at SE(i)
2. Move x from SE(j) to SE(i)
3. Place F on CE(k)
4. Compute F(x) at CE(k)
5. Move y to L
6. Release allocated space

Storage Element (SE) Compute Element (CE)
21
Data Placement (DaP) is an integral part
ofend-to-end functionality
Space management and Data transfer

22
Client
Application
DAGMan
Planning
Stork
Condor-G
GateKeeper
RFT
NeST
StartD
23
How to deliver a common (best
practice)middleware suiteto thecommunity?
24
Best practice

• Requires a true collaboration between all
  parties
• Requires extensive testing
• Requires deployment and adoption
• Requires evaluation metrics
• Requires open minds

25
Common Middleware

• Requires robust implementation.
• Requires professional support.
• Requires continuity and longevity.
• Requires willingness to resolve show stoppers.
• Requires commitment to the concept.
• Requires an exit strategy.

26
TheVirtual Data Toolkit(VDT)is an attemptto
deliver such a middleware suite
27
What is VDT?

• Part of the GriPhyN and VDT projects
• Contributions from many middlewae providers
  (Condor, Globus, PACMAN, EDG, PPDG, )
• Extensive testing in semi-production conditions
• Close interaction with the experiments
• Support infrastructure
• Adopted by key players

28
Standard middleware?

• adjective
• 1. Serving as or conforming to a standard of
  measurement or value.
• 2. Widely recognized as a model of authority or
  excellence a standard reference work.
• 3. Acceptable but of less than top quality a
  standard grade of beef.
• 4. Normal, familiar, or usual the standard
  excuse.
• 5. Commonly used or supplied standard car
  equipment.
• 6. Linguistics. Conforming to established
  educated usage in speech or writing.