The Port d'Informaci Cientfica PIC: Support for dataintensive collaborative science

1
The Port d'Informació Científica (PIC) Support
for data-intensive collaborative science

• Prof. Manuel DelfinoDirector, Port dInformació
  CientíficaPresentation at IEEC Seminar10
  November 2004

2
Outline

• A few words about the so-called e-Science
• PIC A new concept for a support center for
  science embedded in a worldwide environment
• Basics of virtualized computing in comparison to
  single node and cluster computing
• Current key building blocks and the resulting
  LCG-2/EGEE-0 Grid Infrastructure
• Issues for Grid Infrastructure and Resource
  centers in massive data-centric environments
• Questions and comments

3
e-Science or...

• Better Science and Innovation through
  collaborations supported by digital
  infrastructures
• Emerging due to the combination of
• All Scientific and Technological Instrumentation
  becoming digital
• The Internet being ubiquitous and with
  ever-increasing capacity at reasonable cost
• New (and old!) developments in Computer Science
  and Engineering

4
The basic knowledge-generation loop
Data
5
A striking example of digitalization
UDIAT Medial Image Unit at Parc Taulí Hospital
The past
The present and future
6
How is the future of Digital Mammography being
generated?

• Example Dear-Mama (Detection of Early Markers in
  Mammography, see http//xray.ifae.es)IFAE
  Physics Institute Centro Nacional de
  Microelectrónica UDIAT-Parc Taulí Universität
  WienParis Hôpital dEnfants Armand
  Trousseau(Spanish company in process of
  negotiation with the EU)

7
Storing data is not sufficient. Data Management
and Optimized Access must be provided
Dear-Mama(UE FP5)IFAE (coordinador)Centro
Nacional de Microelectrónica UDIAT-Parc
TaulíUniversidad de VienaHôpital dEnfants
Armand Trousseau de Paris
8
The CERN Large Hadron Collider The largest data
producer of this decade (?)
(CMS
experiment)
4 experiments10 Petabytes/year to be analyzed
by worldwide community during gt10 years
Data Recording Offline Analysis
100 MB/sec 2 Petabytes/year
9
Data accumulation expected for the LHC
10
Providing Data Management and Access is not
sufficient!One must Enable Virtual Communities
to collaborate
Data
11
PIC

• Support Center for Collaborative Data-Intensive
  Science through deployment of advanced/innovative
  technologies
• Storage, Management and High Throughput
  Processing of Terabytes to Petabytes
• e-Infrastructures, mainly Grid, to enable
  inter-institutional, trans-national and
  multi-discipine collaborations

12
PIC personnel the most important asset
2 Collaboration Agreement Telefónica ID
13
PIC Family photo 2004
14
Second most important asset Institutional
Stability and Commitment
Multi-annual agreement includes personnel,
equipment and maintenance
15
PIC is housed in part of UAB Edifici D,
reconverted with an investment of 0,3 M
16
Edifici D Built 1990 to house IBM EARN
Supercomputer
17
500 KVA Diesel Generator chilled water huge
air fans
18
PIC Machine Room
UAB Machine Room
PIC Machine Room (early 2003)
Offices
Corridor
19
Offices just across the corridor
UAB Machine Room
PIC Machine Room
Offices
Corridor
20
PIC Safety features
Global Laser-based smoke detection
Potassium Carbonate powder fire extinguishing
bombs
21
PIC Electrical safety features
200 KVA of capilarized electricity
Individual Rack Smoke Detectors connected to
Electrical Breakers
22
PIC Advanced Technology Deployment

• Storage, Management and High Throughput
  Processing of Terabytes to Petabytes
• Storagetek Tape Robot
• 6000 slots x 200 GB/slot (-gt500 GB/slot -gt 1
  TB/slot)
• Extremely reliable high-speed tape drives
• Terabytes of disk cache
• Tape is never accessed directly, always via
  cache.
• Collaboration with CERN on Hierarchical Storage
  Management software tuned for data-intensive
  science
• Cluster of Data Transformers
• Hundreds of PCs Intel/Linux
• Tuned for high data throughput (200 GB disk
  dual GbE)
• Fully switched GbE LAN (-gt 10 GbE over copper)

23
PIC 2 Areas Racks Robot(s)
1000 u of Racks ready for CPU and disk
serversLAN All switched Gbps EtherWAN
Dedicated VLAN 500 Mbps
6000 slot STK L5500 Robot5 9940B drives (200
GB/cartridge gt30 MB/s) Space for 3 more Robots
with virtually no construction work.
24
Transformadores de Datos, Servidores de Disco y
Cinta, Virtualizadores de Servicios
120 Dell P4 3 GHz HT
Grid UI, RB, BDII, SE, CE
Pizza TeraBrik Disk Servers
Dell Poweredge Tape Servers
25
PIC Advanced Technology Deployment

• e-Infrastructures, mainly Grid and high-speed
  tuned WAN
• Separate connection to Anella Científica500 Mbps
  VLAN (-gt1 Gbps-gt10 Gbps)
• Active participation in LCG and EGEE projects
• Virtualization of Data Store and Data
  Transformers
• Location independent authentication through X.500
  PKI worldwide mutual-trust infrastructure
• Groupings of users into Virtual Communities

26
From physical to virtualized computing (1)

• Classical physical computer
• Hardware
• Central Processing Unit
• Memory
• Storage
• Input Device(s)
• Output Device(s)
• Operating System
• Fetch information from Input Device or Storage
  (including itself bootstrapping)
• Within this information, find instructions and
  data to feed the CPU via the Memory
• Send selected information to Output Device or
  Storage

27
From physical to virtualized computing (1)

• Classical physical computer
• Hardware
• Central Processing Unit
• Memory
• Storage
• Input Device(s)
• Output Device(s)
• Operating System
• Fetch information from Input Device or Storage
  (including itself bootstrapping)
• Within this information, find instructions and
  data to feed the CPU via the Memory
• Send selected information to Output Device or
  Storage

Note that this model covers everything from the
first computers (called mainframes using paper
for input and output) through the PC, the
Playstation, the Palm Pilot. QuestionDoes it
cover the iPOD?
28
From physical to virtualized computing (2)

• Multi-user computer
• Same hardware (more of it)
• Operating System
• Allow groupings in the interactions with input
  and output and the contents of memory and
  storage.
• Each grouping corresponds to a user or job.
• Use some algorithm to alternate between the
  groups, feeding instructions and data to the CPU
  and handling input and output as before.
• Note that this alternation (scheduling) is not
  free, hence the total power available to users is
  not 100

29
From physical to virtualized computing (2)

• Multi-user computer
• Same hardware (more of it)
• Operating System
• Allow groupings in the interactions with input
  and output and the contents of memory and
  storage.
• Each grouping corresponds to a user or job.
• Use some algorithm to alternate between the
  groups, feeding instructions and data to the CPU
  and handling input and output as before.
• Note that this alternation (scheduling) is not
  free, hence the total power available to users is
  not 100

• Historical notes
• This was the beginning of the operating system
  within an operating system or hypervisor
  (today at the heart of IBM Z-series servers)
• Demands rapidly outgrew hardware, leading to the
  invention of virtual hardware (memory stored on
  disk is Virtual Memory)

30
From physical to virtualized computing (3)

• Many developments have followed
• For historical completeness I mention Shared
  Memory multi-CPU under one operating system or
  hypervisor
• Another path leads to start interconnecting
  physical computers using networks
• Complexes of mainframes One computers input
  device is really one end of the network, the
  other end connected to another computers output
  device. Important because it is the beginning of
  Device Virtualization.
• Clusters 2 to n computers interacting via a
  network.
• Further device virtualization ? Network File
  Systems, telnet
• Process-to-process interaction across computers ?
  Message Passing Interfaces, cluster
  supercomputers
• Most important for me Virtualization of Access
  Control

31
From physical to virtualized computing (4)
Device Driver
CPU a Process x
Device Driver
CPU b Process y
CPU d Process w
Device Driver
Device Driver
Device Driver
Shared (or virtualized) access control
32
From physical to virtualized computing (5)

• In parallel to these developments
• Moores law gives us more transistors, so CPU
  power and memory grow while prices drop. This
  means the operating system can be more and more
  complicated, can have more virtualization, while
  keeping the user happy.
• Electronic and optical communication technology
  develops faster than any other digital
  technology.
• TCP/IP, DNS and other protocols are organized and
  combined with hardware, fibers and cables to
  build a seamless global network infrastructure
  the Internet.
• In an ideal world, the concept of LAN and WAN
  would have become blurred ? Metacomputing.
• In a less ideal world, we must do more to keep
  distributed computing secure ? The Grid.

33
From physical to virtualized computing (5)
Wait a minuteThis guy is crazy!He did not
mentionthe World Wide Web!!!!Why??????

• In parallel to these developments
• Moores law gives us more transistors, so CPU
  power and memory grow while prices drop. This
  means the operating system can be more and more
  complicated, can have more virtualization, while
  keeping the user happy.
• Electronic and optical communication technology
  develops faster than any other digital
  technology.
• TCP/IP, DNS and other protocols are organized and
  combined with hardware, fibers and cables to
  build a seamless global network infrastructure
  the Internet.
• In an ideal world, the concept of LAN and WAN
  would have become blurred ? Metacomputing.
• In a less ideal world, we must do more to keep
  distributed computing secure ? The Grid.

34
From physical to virtualized computing (6)
Adapted by permission from Ian Foster, University
of Chicago and US Argonne National Lab

• How about access computing resources like we
  access Web content?
• We have no idea where a website is, or on what
  computer or operating system it runs
• Two interrelated opportunities
• 1) Enhance economy, flexibility, access by
  virtualizing computing resources
• 2) Deliver entirely new capabilities by
  integrating distributed resources

35
Key components of a Grid Infrastructure (1)

• For a reasonable price we can have so much
  processing power, storage capacity and high
  quality networking that we can create a Grid
  Infrastructure and Virtualize Everything (even
  the user).
• Note These resources are NOT those you think of
  as helping you process your data and do your
  analysis. I am talking only about a set of
  resources that maintain the virtualization. These
  correspond to the evolution of the overhead
  that we mentioned when discussing multi-user
  computers.
• The actual data processing and computing engines
  are in Resource Centers connected to/by the Grid
  Infrastructure.
• Two key issues to create a secure and useful Grid
• Break the historical model of basing security on
  physical location
• Externalize to the Infrastructure the task of
  keeping track of the state of the resources,
  therefore creating a dynamic, reconfigurable
  distributed computing system

36
A cartoon to illustrate...
37
Key components of a Grid Infrastructure (1)
Applications Delivery
Application Services Distribution
Servers Execution
Source The Grid Blueprint for a New Computing
Infrastructure (2nd Edition), 2004
38
Infraestructura Grid Punto de vista del usuario
Infraestructura Grid
CientíficosColaborando
Middleware específico
39
Una Infraestructura Grid permite la existencia de
muchas Grids
Infraestructura Grid
Middleware general
Middleware específico
Middleware específico
Middleware específico
Comunidades de científicos de diversos campos que
colaboran a distancia
Tejidos Computacionales utilizados de manera mas
flexible y eficiente
40
Key components of a Grid Infrastructure (2)

• What are user and resource virtualization?
• A user (and a resource) on the Grid is identified
  by an X.509 Certificate using a technology known
  as Public Key Infrastructure or PKI. This, in
  some sense, replaces the old username/password
  for a user and the IP address for machines.
• But a user who only gets a certificate cannot use
  any (virtual) resource on the Grid. She must
  first become a member of one or more Virtual
  Organizations.
• Similarly, a machine in a Resource Center which
  only has a certificate cannot be seen by any
  (virtual) user. For that, it must be configured
  to accept work from a Virtual Organization.
• The same physical user can have different roles
  on the Grid.
• Physical resources can be dynamically
  reconfigured to serve the varying needs of VOs.
• Without user virtualization we have an
  interesting variant called Utility Computing
• The other extreme, where the user is
  virtualized but not the resources, roughly
  corresponds to Peer-to-Peer Computing

41
Key components of a Grid Infrastructure (3)

• This is starting to sound complicated. Can this
  really be built?
• Yes.
• After almost 10 years of RD we had components
  from the Globus team, the Condor team, the EU
  DataGrid project, etc.
• Each component is like a piece of the puzzle that
  if assembled together correctly would Virtualize
  Everything.
• This involves creating and updating a giant,
  distributed, redundant database (in fact using
  another important technology called Lightweight
  Directory Access Protocol or LDAP)
• Initially, each project had decided its own
  database structures, and therefore the components
  did not interoperate.
• The Large Hadron Collider community helped to
  catalyze an agreement on a common database schema
  (called GLUE !)
• This allowed the deployment of the first large
  scale Grid LCG-1
• A year later we have an improved and quite robust
  LCG-2

42
Key components of a Grid Infrastructure (4)

• Virtualized resources ? Rough equivalent
• CE Computing Element ? Computer
• RB Resource Broker ? Batch system
• UI User Interface ? Interactive computer
• SE Storage Element ? Disk directory
• RLS Replica Location Service
• VO Server
• Also need to have
• worldwide distributed user registration
• worldwide X.509 certificate issuers
• interoperable certificate validity info

Source LCG-2 User Guide
43
Key components of a Grid Infrastructure (4)

• The virtualized configuration is maintained by
  LDAP servers using a Berkeley Database engine
• Since datasets may be replicated on the Grid, we
  need
• A way to uniquely identify a dataset (the Grid
  Unique ID)
• A way to keep track of the replicas (the Replica
  Catalog)
• A way to virtualize the replicas (LFN?GUID?SURL
  mapping)

Source LCG-2 User Guide
44
Key components of a Grid Infrastructure (5)

• How about access computing resources like we
  access Web content?
• We have no idea where a website is, or on what
  computer or operating system it runs
• Two interrelated opportunities
• 1) Enhance economy, flexibility, access by
  virtualizing computing resources
• 2) Deliver entirely new capabilities by
  integrating distributed resources

Adapted by permission from Ian Foster, University
of Chicago and US Argonne National Lab
45
The Enabling Grids for E-sciencE Project

• The EGEE Project
• Scalability of deployment, operation, monitoring
  and optimization achieved through a Federated
  Scheme
• LCG-2 ?EGEE-0
• Grid Infrastructure is opened for use by all
  sciences
• Add non-LCG Resource Centers
• Organize maintenance of non-LCG Virtual
  Organizations
• Certificate Authorities usually organized by
  country
• Funded for 2 years by the EU
• Proposing additional 2 years

46
From LCG-2 to EGEE-0 Grid Infrastructure
becomes a reality
INTA-CAB Centro de AstrobiologíaCNB Centro
Nacional de Biotecnología(see http//goc.grid-sup
port.ac.uk/gppmonWorld/gppmon_maps/lcg2.html )
47
Proyectos concretos de Ciencia e Innovación ya en
marcha

• Física de Altas Energías
• Preparación para el proceso de datos del Large
  Hadron Collider
• Análisis de datos del telescopio MAGIC situado en
  La Palma, Canarias
• Análisis de datos del experimento CDF del
  laboratorio Fermilab, Chicago, EEUU
• Análisis de datos del experimento K2K de
  oscilación de neutrinos en Japón
• Salvaguarda de simulaciones del futuro
  experimento AMS, que volará en la Estación
  Espacial Internacional
• Medicina/Salud
• Construcción de base de datos de imagen médica
  con la UDIAT del Hospital Parc Taulí, Sabadell
• Salvaguarda de simulaciones internacionales marco
  de EGEE

48
Issues for experimental particle physics (1)

• LHCb experiment worldwide simulation using
  mixture of Utility and Grid185 K jobs185 M
  events61 TBytes3.7 Mhours CPU

Source Report by UB and USC groups in LHCb
49
Issues for experimental particle physics (2)
ATLAS Data Challenge II running on LCG-2 Resource
Centers
Source Report by ATLAS Collaboration at LCG
meeting
50
Issues for experimental particle physics (4)
Source Report by José Hernández, CIEMAT at CMS
Data Challenge 04 Review
51
Issues for experimental particle physics (4)
files
MBytes

• Irregular data transfer
• Transfer Agent kept up with data availability
• Typically lt 10 MB/s
• Files very small
• Network stress test on May 1st

March
May
April
Total files 446652 Total data
volume 6049963 MBytes
Source Report by José Hernández, CIEMAT at CMS
Data Challenge 04 Review
52
Issues for experimental particle physics (4)

• Data available at EB lt 10 MB/s. Average file size
  small
• On May 1st network stress test transferring big
  zip files to T1s (up to 1.6 GB)
• 768 GB (3345 files) transferred to PIC in 10
  hours on May 1st
• 119 GB ( 338 files) transferred to PIC in 1
  hour, 20 minutes on May 2nd
• Typical transfer rate 30 MB/s

Source Report by José Hernández, CIEMAT at CMS
Data Challenge 04 Review
53
Issues for experimental particle physics (4)
Analysis
T2
GDB
T1
EB
Reconstruction
Transfer and replication agents
Drop and Fake Analysis agents
Publisher and configuration agents
EB agent
Source Report by José Hernández, CIEMAT at CMS
Data Challenge 04 Review
54
Issues for experimental particle physics (4)
Source Report by José Hernández, CIEMAT at CMS
Data Challenge 04 Review
55
Collaborative Science enabled by global
e-Infrastructures
ATLAS physicists, Tokio, Japan
Resource Centers accepting ATLAS, CMS, LHCb and
MedIMG Virtual Organizations
LHCb physicists, Santiago, Spain
Bellaterra, ES
RAIM
Sabadell, Spain
Resource Centers accepting LHCb VO in Canada,
Japan, Portugal, Spain, Taiwan, USA
56
Excellent Infrastructure and Process Engineering
the key to success of Grid
Source Gartner Group, April 1999
57
Economies of Scale in Grid Computing
An example from the development of electrical
power from a cottage industry to a dependable
infrastructure

• Decouple production
• consumption, enabling
• On-demand access
• Economies of scale
• Consumer flexibility
• New devices

Quality, economies of scale
Time
Adapted by permission from Ian Foster, University
of Chicago and US Argonne National Lab
58
Virtual Organizations Large or small, few or
many?
More users, new markets
2020
Commoditization
2010
2000
1990
Decade
1980
Time-sharing computing
1970
Operating systems
Numerical methods
1960
Mainframes
Programming
1950
Computers
1
10
100
1000
10000
100000
1000000
10000000
Complexity and Functionality
59
Virtual Organizations Large or small, few or
many?
We are here
WWW invented here
Internet born here
First e-mail ever sent
60
The EGEE Vision

2003 2004 2005 2006 2007 Year 1 Year 2 Year 3
Year 4
Applications
Resource Centres

• EGEE Project 70 partners, 32 M of financing
  2004-2005
• PIC regional operations centre in Southwest
  Europe

61
Thank you. Questions or Comments?
62
Additional slides
63
Comisión Gestora del PICOctubre 2004