Introduction to EGEE and gLite middleware

1
Introduction to EGEE and gLite middleware
Giuseppe Andronico April 18, 2009 --
Taipei INFN Sez. CT
2
Contents

• EGEE status, history and perspectives
• Grid services gLite 3.1
• Introduction to basic systems
• Main components
• Component interaction
• Sources of further information

3

• Quality
• Monitoring via Nagios - distributed via official
  releases, configured through YAIM, integrated
  with other tools
• Gradual implementation of Service Level Agreement
  with sites
• Result 85 of sites are now above the 75
  availability threshold
• Geographical expansion
• now have production sites all across Asia
  Australia, China, India, Japan, Korea, Malaysia,
  Pakistan, Taiwan, Thailand
• In certification Indonesia, Philippines, Vietnam
  

280 sites 45 countries gt80,000 CPUs gt20
PetaBytes gt14,000 users gt250,000 jobs/day
4
Production Volunteer grids
5
Applications database

• 61 registered applications so far
• Joint initiative between EGEE and SEE-GRID-SCI
• http//appdb.eu-egee.org/ http//grid.ct.infn.it
  /egee_applications/

Readiness status 7 HandledByRS 6 In production
5 Ready for deployment 4 Ready for validation
3 Ready for portal interface 2 Ready for gLite
middleware 1 Ready for standalone use / running
in a local cluster
6
EGEE?EGI transition planning
7
Grid Security
Users in many locations and organisations
Access services (User Interface) logon, upload
credentials, run m/w commands
Built on Grid Security Infrastructure
(GSI)Encryption and Data Integrity,
Authentication and Authorization
Gate Keeping Identify users and give permissions
Data Storage
High Level Service
Computation
Software
Resources in many locations and organisations
8
Introductive concepts

• Computational Resources machines where an user
  can execute a program (job), store or retrieve
  files
• Job is a computational task (an executable, a
  script) that the user wants to be executed in the
  Grid, having finally the results back on it's
  machine
• VO people owning at different institutions but
  working at similar problems sharing resources,
  applications, data,

9
Grid Concepts
10
Extendable Infrastructure
11
gLite decomposition
Access
CLI
API
Security Services
Information Monitoring
Services
Authorization
Information Monitoring
Job Monitoring
Auditing
Authentication
Data Services
Job Mgmt. Services
MetadataCatalog
JobProvenance
PackageManager
File ReplicaCatalog
Accounting
StorageElement
DataMovement
ComputingElement
WorkloadManagement
Overview paper http//doc.cern.ch//archive/electro
nic/egee/tr/egee-tr-2006-001.pdf
12
Work Load Management
Users

• Tools to
• Submit jobs to a CE
• Monitor jobs
• Get outputs
• Transfer files to CE
• Transfer files between CE and SE

How do I run a job on a Compute Element (CE) ?
(CE batch queue)
Resources
Compute elements
Data storage
Network resources
13
WMS
14
Information System (IS)
Users

• Resource publishes
• Configuration
• Latest status
• Access information

Which resources can I use?
Resources
Compute elements
Data storage
Network resources
15
Information System
User applications WMS Monitoring services
Hierarchical information
VO
Site
Resource
16
Data Management
Users

• Data Management provides
• Storage Space
• Data Transfer
• Replication

How can I manage and share my data?
My data are in files, and Ive terabytes
Our data are in
Resources
Compute elements
Data storage
Network resources
17
LCG File Catalog (LFC)
Catalogs

• LFN (Logical file name)
• GUID (Grid unique identifier)
• SimLinks
• SURL (Site URL)
• TURL (Transfer URL)

18
Data Movement

• GridFTP basic transfer protocol
• Data Scheduler (DS)
• File Transfer/Placement Service (FTS/FPS)
• Transfer Queue (Table)
• Transfer Agent (Network)

19
Basic gLite use caseJob submission
Information System
Submit job (executable small inputs)
query
Retrieve status (small) output files
create proxy
query
publish state
Submit job
Retrieve output
Job status
Logging
Register file
Input file(s)
Job status
process
VO Management Service (DB of VO users)
Output file(s)
Logging and bookkeeping
20
Main components
User Interface (UI) The place where users
logon to the Grid
Resource Broker (RB) (Workload Management System
(WMS) Matches the user requirements with the
available resources on the Grid
Information System Characteristics and status of
CE and SE
File and replica catalog Location of grid files
and grid file replicas
Logging and Bookkeeping (LB) Log information of
jobs
Computing Element (CE) A batch queue on a sites
computers where
the users job is executed
Storage Element (SE) provides (large-scale)
storage for files
21
Main components
User Interface (UI) The place where users
logon to the Grid
Resource Broker (RB) (Workload Management System
(WMS) Matches the user requirements with the
available resources on the Grid
All built upon authorisation, authentication,
security
Information System Characteristics and status of
CE and SE
File and replica catalog Location of grid files
and grid file replicas
Logging and Bookkeeping (LB) Log information of
jobs
Computing Element (CE) A batch queue on a sites
computers where
the users job is executed
Storage Element (SE) provides (large-scale)
storage for files
22
Who provides the resources?!
23
Further information

• EGEE digital library http//egee.lib.ed.ac.uk/
• EGEE www.eu-egee.org
• gLite http//www.glite.org/
• GILDA https//gilda.ct.infn.it /
• Open Grid Forum http//www.gridforum.org/
• Globus Alliance http//www.globus.org/
• VDT http//www.cs.wisc.edu/vdt/

24
Questions