Particle Physics Data Grid Scientific Discovery through Advanced Computing Collaboratory Pilot, Tril - PowerPoint PPT Presentation

Loading...

PPT – Particle Physics Data Grid Scientific Discovery through Advanced Computing Collaboratory Pilot, Tril PowerPoint presentation | free to download - id: 3a42b-Mjk0M



Loading


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation
Title:

Particle Physics Data Grid Scientific Discovery through Advanced Computing Collaboratory Pilot, Tril

Description:

co-chair of the HICB-Joint Technical Board ... Globus/ISI, Carl Kesselman, Ann Chervenak ... Steve Chan, Steve Lau , Craig Tull, Matt Crawford, David Bianco, Hao Wang ... – PowerPoint PPT presentation

Number of Views:99
Avg rating:3.0/5.0
Slides: 26
Provided by: rober492
Category:

less

Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: Particle Physics Data Grid Scientific Discovery through Advanced Computing Collaboratory Pilot, Tril


1
Particle Physics Data Grid Scientific Discovery
through Advanced Computing Collaboratory Pilot,
Trillium High Energy Physics Intergrid
Coordination and Joint Technical Boards
  • LHC Review, January 14th
  • R. Pordes, Fermilab,
  • a PPDG Coordinator
  • co-chair of the HICB-Joint Technical Board

http//www.ppdg.net/docs/presentations/PPDG_HICB_0
11403.ppt
2
  • PPDG
  • Introduction
  • Management structure, coordination with the
    experiments
  • Status and deliverables
  • Prospects
  • Trillium
  • Introduction,
  • Prospects
  • HICB and HICB-JTB
  • Process, Goals, Contacts with the Experiments
  • Status and deliverables
  • Prospects
  • www.ppdg.net
  • Q3-02 Qtrly Report
  • Year 2 Plan 
  • Analysis Tools meeting (12.02)
  • Site Authentication and Authorization Report
    (12.02)
  • Joint Collaboration Meeting
  • Newsletter
  • US-EU Demos
  • www.hicb.org
  • www.hicb.org/jtb
  • www.hicb.org/GLUE

3
Reference urls
  • www.ppdg.net
  • Q3-02 Qtrly Report - http//www.ppdg.net/docs/ppdg
    _qtrly_jul-sep-02.pdf
  • Year 2 Plan  - http//www.ppdg.net/docs/Project/PP
    DGYear2.doc
  • Analysis Tools meeting (12.02) -
    http//www.ppdg.net/mtgs/19dec02-cs11/
  • Site Authentication and Authorization Report
    (12.02) - http//www.ppdg.net/mtgs/19dec02-siteaa/
  • Joint Collaboration Meeting - http//www.ivdgl.or
    g/events/view_agenda.php?id3
  • Newsletter - http//www.hicb.org/TrilliumNewslette
    r/
  • US-EU Demos http//www.ivdgl.org/demo/
  • www.hicb.org
  • www.hicb.org/jtb
  • www.hicb.org/GLUE
  • http//www.hicb.org/glue/glue-v0.1.3.doc

4
Introduction to PPDG
  • DOE SciDAC Collaboratory Pilot. Currently in the
    middle of 2nd year of 3 year current program.
  • PIs Miron Livny, Richard Mount, Harvey Newman.
    Executive Team Doug Olson, Miron, Ruth
  • Annual budget 3M for effort - no funding for
    hardware. Funding split between MICS and HENP
    offices.
  • Computer Science Groups
  • Globus
  • Condor
  • SDSC
  • LBNL SRM
  • 1-3 FTEs each.
  • Running Experiments
  • Nuclear Physics Star, JLAB 1 FTE each
  • HEP BaBar, D0 2-3 FTE each
  • LHC Experiments ATLAS, CMS 2-3 FTE each
  • LQCD connection JLAB

5
PPDG - Scope and Goals - the Green Grid
  • Deploy End-to end-Experiment application Grids -
    work and integrate with the local environment.
  • Integrate common grid technologies and standards
    to have working end-to-end systems - ensure
    useful work.
  • Development integration Projects are peered
    collaborations of 1 CS and 1 Experiment group -
    with exceptions to prove the rule.
  • Cross cut technology Common Services
    strategically defined to encourage
    cross-experiment and cs group discussion and
    adoption.
  • Leverage wider efforts - think globally, act
    locally.
  • Technical Challenges - Deployment is different
    than demonstration
  • For the running experiments constrained to mesh
    with legacy systems Need to maintain operational
    current data handling and even Distributed Data
    Managements infrastructures throughout the
    lifetime of the project allow only tiny steps to
    be taken and even those take a long time.
  • For LHC experiments and LQCD data handling models
    are still being developed in large
    collaborations. Parallel solutions and divergence
    easy.
  • Sociological Challenges of course.

6
PPDG Work Areas
  • Data Movement focus includes Reliability and
    Robustness up front.
  • Experiment Production - simulation,reconstruction
    - Grids were an early primary deliverable -
    only at 10 because RELY on CS-1 through CS-9.
  • Steering Committee extends the areas of work as
    we need to.
  • Focus is moving down the list as the project
    proceeds.
  • Clearly PPDG is not developing all these
    capabilities. We discuss and address application
    needs and technology extensions in all these
    areas.

7
Who participates
  • Globus/ANL, Ian Foster , Mike Wilde , Jenny
    Schopf , Willliam Alcock, Von Welch, Stu Martin
  • ATLAS, John Huth, Torre Wenaus, L. Price , D.
    Malon , A. Vaniachine, E. May , Rich Baker , Alex
    Undrus  , Dave Adams, Wengshen Deng, G.
    Gieraltowski , Dantong Yu
  • BaBar, Richard Mount, Tim Adye , Robert Cowles ,
    Andrew Hanushevsky , Adil Hassan , Les Cottrell,
    Wilko Kroeger
  • CMS, Lothar Bauerdick, Harvey Newman , Julian
    Bunn , Conrad Steenberg , Iosif Legrand ,
    Vladimir Litvin, James Branson , Ian Fisk, James
    Letts, Eric Aslakson, Edwin Soedarmadji, Saima
    Iqbal, Suresh Man Singh, Anzar Afaq, Greg Graham
  • D0, Lee Leuking , Igor Terekhov , Andrew
    Baranovski, Gabriele Garzoglio, Siddharth Patil,
    Abhishek Rana, Hannu Koutaniemi, John Weigand
  • HRM/LBNL, Arie Shoshani  , Alex Sim  , JunminGu,
    Alex Romosan
  • SRB/UCSD, Reagan Moore , Bing Zhu
  • JLAB, William Watson , Sandy Philpott, Andy
    Kowalski , Bryan Hess , Ying Chen, Walt Akers
  • STAR , Jerome Lauret, Gabrielle Carcassi, Eric
    Hjort, Doug Olson
  • Condor/U.Wisconsin, Miron Livny , Paul Barford ,
    Peter Couvares, Rajesh Rajamani, Alan DeSmet,
    Alain Roy,Todd Tannenbaum
  • Globus/ISI, Carl Kesselman, Ann Chervenak
  • SiteAAA(if not listed above), Dane Skow, Bob
    Cowles, Booker Bense, Tomasz Wlodek , Igor
    Mandrechenko, Steve Chan, Steve Lau , Craig Tull,
    Matt Crawford, David Bianco, Hao Wang

8
PPDG Management and Day to Day Operation
  • PPDG funded effort is managed by Team Leads (or
    designees), in collaboration with the site or
    project leaders. Well integrated into their wider
    teams.
  • Experiment Team Leads are Computing Managers.
    Computer Science Team Leads are CS group leaders.
  • Project Business conducted through weekly
    Executive and Collaboration phone meetings
    review common service areas e.g. file replication
    and cover end-to-end status of experiment
    applications. 2 collaboration several focus
    meetings a year
  • Information Flow through comprehensive quarterly
    reports, mail lists, web pages
  • Steering Committee phone meetings monthly and
    face to face meetings 2 per year.
    Construct, review and endorse yearly plan. Take
    decisions on budget and strategy.
  • Outreach and Liaison get much attention travel,
    coordination, communication and outreach to peer
    projects. Liaisons from iVDGL, GRiPhYN, LCG,
    International etc.

PIs
Steering Committee PIs, Executive Team, Team
Leads, Liaisons
  • Executive Team

9
Where PPDG Experiment Effort is focussed
  • Experiment highest priority should be directly
    related to development and support for the
    integration and delivery of working grid
    applications in the experiment - e.g. Anzar Afaq
    in CMS is the coordinator of the Integration
    Production Testbed Gerry Giertowski in ATLAS
    evaluated and integrated European Data Grid
    software with production interface.
  • Computer Science highest priority should be
    directly related to development and support for
    the identified short term requirements of the
    experiment applications. e.g.Globus changes to
    CAS in response to SiteAA requirements Condor
    extensions to DAGMAN to support CMS production
    testbed
  • Development are most appropriate in areas where
    existing or emerging grid technologies are
    lacking, and the experiments have short to medium
    term needs. E.g. CMS and ATLAS developments for
    physics analysis SRB support for federations
  • Outreach Work with projects and standards bodies
    e.g. Storage Management Interface definitions
    e.g.SRM Vx. work with all on proposing
    development of missing capabilities for
    production grids. e.g. with DOE SG on US CA/RA.

10
Status and Deliverables
  • All experiments have deployed applications using
    Grid Data Movement technology for the benefit of
    their collaborations
  • Data transfer GridFTP, BBCP SRM(DRM),
  • Data movement management SAM, SRB, MAGDA,
    GDMP,
  • Most experiments have prototype/development
    applications with Grid Job Scheduling and
    management Condor-G job scheduling to multiple
    Globus-Gatekeepers and Sites Resource discovery
    and monitoring through MDS DAGMAN for job flow.
  • Cross project additional effort in Authentication
    and Authorization yielding working solutions
    where initially there were none.
  • Many specific developments from experiment
    engineers to achieve working applications.

11
Year 2, Year 3,.. deliverables
  • File Transfer, Replication and Data Management -
    in particular continued integration and
    deployment in the applications.
  • Scheduling and management of physics data
    processing and analysis applications. This brings
    with it the need to address the details of
    intelligent specification, request, execution and
    presentation of work.
  • Authorization - how application resource
    requests coming from the grid get mapped to the
    allocation and use of local facility resources,
    how the actual use is reported back to the
    authorizing entity.
  • Production Capabilities Integration and
    Deployment, System Operation, Qualities for
    Production
  • Major attention in year 3 Physics Analysis over
    Grids

12
Analysis Requirements Prototyping
Remote PROOF Cluster
Local PC
stdout/obj
root
ana.C
proof
.root
TFile
node1
ana.C
proof
.root
GRID as black box
TNetFile
node2
proof
TFile
.root
Select and extract data from grid, Analyze using
local resources without Further interaction with
grid
node3
proof
TFile
.root
node4
13
An Issue for all..
Maintenance and Support
  • PPDG statement is that Maintenance, Operations
    and Support is outside the Project Scope.
  • Software Maintainenance and support belongs with
    each Development group ie within the Experiment
    infrastructure or with the Computer Science
    Project.
  • Issue of longevity of Computer Science Groups an
    important issue which PPDG helps lobby for.
  • Operations is the responsibility of each
    experiment or application group - outside of the
    PPDG project.

14
PPDG prospects - current and future
  • Provides natural path through well established
    and recognised computer science groups for
    deliverables to be disseminated and used by the
    wider community.
  • Provides natural forum for common technology
    components and capabilities to be discussed,
    developed and shared across experiments and
    applications.
  • As we move into the final year of the SciDAC
    Pilot we see PPDG Phase III (phase I- NGI phase
    II-SciDAC) necessary to develop the higher layers
    of the application and production GRID and
    Application services
  • HENP Data Analysis include Interactive
    capabilities, support for Physics Objects and
    Methods etc.
  • Multi-level Security Authorization and Accounting
  • Data, Meta-Data and Code Provenance and
    Management.
  • Robustness and Fault tolerant Services
  • Adoption and Testing of Emerging Technologies
    e.g. Web services.

15
Observations
  • Running experiments progress are impacted by the
    need to support data taking and ongoing data
    analysis.
  • CS groups get split 6 ways - which makes the
    targetted effort for each experiment small.
  • The steering committee has endorsed increased
    collaboration with EDG. Several experiments are
    prototyping/integrating with EDG releases.
  • The steering committee has added direct
    collaborations with LBNL-DMF (performance
    monitoring) SLAC-IEPM (netowork monitoring).
  • PPDG benefits greatly from leveraged effort from
    all groups.
  • Collaboration across the multi-dimensional
    project matrix involves measureable effort.
  • Structured Virtual Organization management and
    services, Site/Fabric-Grid interfacing, End to
    End error handlng and Troubleshooting, are
    identified as important issues which require
    increased effort now.
  • PPDG goals help and hinder developing final grid
    solutions early results prove the technology but
    one has to address many/all details at once.

16
Trillium
  • Name for the collaboration between PPDG, iVDGL
    and GriPhyN. It allows us to
  • Make contributions between and across these
    projects without stress of which project is
    doing what
  • Work as a unified and consistent entity with the
    EU and other Grid projects.
  • Have low-overhead collaborative projects e.g for
    Security, Newsletter, Production Grids and
    Demonstrations.
  • Naturally allow the deployment projects to take
    advantage of research projects as technologies
    emerge. E.g. Virtual data in PPDG application
    grids.
  • Remembering that
  • Experiments and CS groups dont completely
    overlap. Time scales are different. The focus
    varies.

17
e.g. SC2002 Demonstrations
18
Trillium Prospects?
  • Continued forum to help the US HEP Grid projects
    form a common view and strategies, lobbying
    group, joint meetings and work.
  • Provide a natural collaboration across DOE and
    NSF projects. Is this of benefit to the funding
    agencies?
  • might hope for some endorsement and backing of
    the approaches for the future....

19
High Energy Physics Intergrid Coordination
  • Open to all participating in HENP grids.
    Participants from EU (specially EDG and DataTAG),
    Asia and US grid projects(Trillium), national
    grid projects (specially GridPP) and experiment
    grid projects (specially LCG)
  • Started at the first Global Grid Forum in
    Amsterdam. Commitment to standards shown by
    decision to co-locate all HICB Meetings with
    GGFs.
  • Has a strong international focus on policies,
    procedures and technologies, cross-project and
    cross-continent joint projects.
  • Past year has shown it to be beneficial as a
    non-competitive forum for discussion and building
    consensus on strategic issues. E.g. Understanding
    of licencing for EDG and US grid middleware
  • Joint Technical board (JTB) provides definition,
    oversight and tracking of joint projects -
    coordinators of various grid projects. Co-chairs
    are Ian Bird - LCG Grid Deployment board area
    manager and myself.

20
Grid Laboratory Uniform Environment (GLUE)
  • Initial Goals defined in Feb 2002 by JTB. to
    address interoperability from the bottom up
    between application and project grid domains.
  • While identical middleware is the shortest path
    to compatability, existing situation has
    different environments and deployments
  • Any Deliverables (code, configuration,
    documents) to be deployed and supported through
    EDG and VDT release structure.Effort mainly from
    DataTAG and LCG in EU and iVDGL, VDT and PPDG in
    US.
  • Technology Demonstrate that each service will
    interoperate.Provide support for cross-continent
    execution of applications. Identify any lacks and
    incompatabilities early.
  • SociologyLearn to work across projects and
    boundaries without explicit mandate or authority
    but for the longer term good of the
    whole.Intent to rapidly expand from 2 continents
    to Global.
  • Strategy interoperability as part of the
    ongoing culture of HENP grid projects. Provide
    input to standards and protocols.

21
Glue and Interoperability status and deliverables
  • Phase I - 60-80 complete in fall 2002
  • GLUE Schema - Common resource information
    definitions presentation across Globus, EDG and
    VDT.
  • Support for cross-domain job submission and
    execution.
  • Cross-trust of Certificate Authorities,
    interoperability of services for authenticaion
    and authorization based on gridmapfiles.
  • Common storage management interface and
    interoperable implementations in the US SRM V1.
  • Phase II just getting underway - schedules range
    through 2003
  • Storage Management standards V2.
  • Release and installation certification and
    testing
  • Software meta-packaging, distribution, and
    configuration
  • Data Movement - currently handled by the
    experiments at the application level

22
(No Transcript)
23
What coordination takes.. people involved in 2002
(none full time)
24
Plans for HICB, JTB
  • Positioning to more actively collaborate with
    Asia.
  • GLUE will continue only as long as participants
    and endorsers deem it a useful vehicle.
  • Usually surprising new areas of fruitful
    discussion at the HICB meetings.
  • JTB becoming a useful umbrella for joint projects
    with in place means for endorsement and buy in
    (manpower always a touchy issue of course).
  • As grid projects come and go a loosely defined
    and self-selected forum will continue to be
    useful.
  • Co-location with GGF is appropriate as ultimate
    success of the grid depends on open and complete
    standards, principles and policies.

25
In Summary - a PPDG view
  • It is useful to have a joint Collaboration
    between experiments already storing and analysing
    up to a Petabyte of data and the LHC experiments
    that are developing and ramping up production and
    preparing for analysis.
  • Contacts between the Experiments and PPDG relies
    on the commitment, energy and constant attention
    of the Team Leads..
  • Success relies on having sufficient Leadership
    and Management involvement.
  • Compromise and acceptance of small steps rather
    than large deliverables seems to give better
    results.
  • This brings a danger of postponing or having
    insufficient mass and energy to address the
    difficult or effort intensive needs.
  • Inclusive and non-competitive loose coordination
    peer groups are a help.
About PowerShow.com