The Open Science Grid OSG

1
The Open Science GridOSG
Ruth Pordes Fermilab
2
What is OSG?

• A Consortium of people working together to
• Interface Farms and Storage to a Grid and
• Researchers using these resources by adapting
  their applications to run on the Grid and
• Software developers providing middleware and
• A project that provides the Operations, Support,
  Training and Help to make it effective.

3
Who is OSG ?

• Large global physics collaborations US ATLAS,
  US CMS, LIGO, CDF, D0, STAR
• Research collaborations such as Mariachi, GROW,
• Grid technology groups Condor, Globus, SRM, NMI
• Many DOE Labs and DOE/NSF sponsored University IT
  facilities and
• Partnerships with TeraGrid and EGEE, and Campus
  Grids such as TACC, GLOW etc.

4
OSG Consortium
5
When is OSG ?

• Grown from of grass-roots collaboration of
  GriPhyN, iVDGL and PPDG participants in 2003.
• 5 years of funding starting 9/2006 from DOE
  SciDAC-II and NSF MPS and OCI
• Deliver to US LHC and LIGO scales in 2008 and
  2009
• Need to routinely distribute data at 1-5 Gbps
  over 10-50 sites.
• Need to routinely exceed 10,000 running jobs per
  client
• Need to reach 99 success rate for 10,000 jobs
  per day submission under heavy load
• Active engagement effort centered at RENCI to
  include new sciences.

6
The OSG Map Aug-2006
7
OSGs world is flat - a Grid of Grids - from
Local to Global
Global Science Community Systems
e.g. CMS , D0
National CyberInfrastructures for Science
Local Campus And Regional Grids
e.g. OSG - TeraGrid
e.g. FermiGrid, NWIC
8
From the Local (Campus) Grids to the Wide Area
Grid

• Within an organization a Local Grid provides for
  simplicity in sharing and efficiency in
  purchasing and administration.
• However, researchers collaborate outside the
  bounds of a single campus.
• With a uniform environment acting globally is
  just an extension of acting locally.
• Open Science Grid focuses on interoperation of
  the local and the wide area.

9

• A resource can be accessed by a user via the
  campus, community or national grid.
• A user can access a resource with a campus,
  community or national grid identity.

10
Example Uses

• High Energy Physics
• ATLAS - gt15 Million proton collision events
  simulated at 10 minutes each
• CMS - gt70 Million events simulated, reconstructed
  and analyzed
• Biology - GADU
• Populates databases from search and analysis of
  similarities and differences among thousands of
  publicly available genome and protein sequences
  and metabolic pathways.
• Gravitational Wave Physics - LIGO Data grid
• Eases grid tools to ensure that 9 computing
  sites have a copy of the interesting data, and
  researchers at 36 LSC institutions use the LDG to
  find the data they need."
• Math research
• Education
• Grid Summer Workshop teaches students to run jobs
  on OSG and TeraGrid.

11
Running (and monitored) OSG jobs in 06/06.
12
Example GADU run in 04/06
13
Integration Testing of the System

• Multi-site Integration Grid tests new OSG
  Releases and Configurations.
• Software Readiness and Validations occur before
  deployment on the Integration Grid.

Integration Grid Sites
14
CMS - US part of a Global Community Grid
OSG
EGEE
CERN
USA
Germany
France
UNL
MIT
Data jobs moving locally, regionally globally
within CMS grid. Transparently across grid
boundaries from campus to global.
15
How do People and Organizations Participate?

• VO Registers with with Operations Center
• Signs VO Agreement
• User registers with VO
• User added to VOMS of one or more VOs.
• VO responsible for users to sign AUP.
• VO responsible for VOMS service support.
• Sites Register with the Operations Center
• Signs the Service Agreement.
• Agree on which VOs to support (striving for
  default admit)
• VOs and Sites provide Support Center Contact and
  join Operations groups

16
Community Documentation..
17
(No Transcript)
18
OSG Principles

• Systems are Virtual Organization scoped -- groups
  of users working together with a shared, common
  environment.
• Sites maintain control, authority and management
  of use of their Grid accessible resources.
• Opportunistically available resources are
  beneficial. Priorities are governed by Policies
  (role based within a VO).
• The distributed system is heterogeneous.
  Information must be provided to allow
  applications to know which resources they can
  successfully use.

19
VO software stacks that Rely on the Virtual Data
Toolkit

LHCServices Framework
OSG VO Framework
BioServices Framework
LIGOData Grid
Applications
OSG Release Cache VDT Configuration,
Validation, VO management
Virtual Data Toolkit (VDT) Common Services NMI
VOMS, CEMon (common EGEE components), MonaLisa,
Clarens, AuthZ
Infrastructure
20
What is the VDT?

• A collection of software
• Grid software (Condor, Globus and lots more)
• Virtual Data System (Origin of the name VDT)
• Utilities
• An easy installation
• Goal Push a button, everything just works
• Two methods
• Pacman installs and configures it all
• RPM installs some of the software, no
  configuration
• A support infrastructure

21
Who uses the VDT?

• Open Science Grid
• LIGO Data Grid
• LCG
• LHC Computing Grid, from CERN
• EGEE
• Enabling Grids for E-Science

22
What software is in the VDT?

• Security
• VOMS (VO membership)
• GUMS (local authorization)
• mkgridmap (local authorization)
• MyProxy (proxy management)
• GSI SSH
• CA CRL updater
• Monitoring
• MonaLISA
• gLite CEMon
• Accounting
• OSG Gratia

• Job Management
• Condor (including Condor-G Condor-C)
• Globus GRAM
• Data Management
• GridFTP (data transfer)
• RLS (replication location)
• DRM (storage management)
• Globus RFT
• Information Services
• Globus MDS
• GLUE schema providers

23
What software is in the VDT?

• Client tools
• Virtual Data System
• SRM clients (V1 and V2)
• UberFTP (GridFTP client)
• Developer Tools
• PyGlobus
• PyGridWare
• Testing
• NMI Build Test
• VDT Tests

• Support
• Apache
• Tomcat
• MySQL (with MyODBC)
• Non-standard Perl modules
• Wget
• Squid
• Logrotate
• Configuration Scripts
• And More!

24
Due diligence to Security

• Risk assessment, planning, Service auditing and
  checking
• Incident response, Awareness and Training,
  Configuration management,
• User access Authentication and Revocation,
  Auditing and analysis. End to end trust in
  quality of code executed on remote CPU
  -signatures?
• Identity and Authorization Extended X509
  Certificates
• OSG is a founding member of the US TAGPMA.
• DOEGrids provides script utilities for bulk
  requests of Host certs, CRL checking etc.
• VOMS extended attributes and infrastructure for
  Role Based Access Controls.

25
Operations Model
Real support organizations often play multiple
roles
Lines represent communication paths and, in our
model, agreements. We have not progressed very
far with agreements yet.
Gray shading indicates that OSG Operations
composed of effort from all the support centers
26
The OSG VO

• A VO for individual researchers and users.
• Managed by the OSG itself.
• Learn how to use the Grid!

27
In Summary

• A production grid is the product of a complex
  interplay of many forces
• Resource providers
• Users
• Software providers
• Hardware trends
• Commercial offerings
• Funding agencies
• Culture of all parties involved

28
Where do you learn more?

• www.opensciencegrid.org
• http//vdt.cs.wisc.edu
• http//osg.ivdgl.org/twiki/bin/view