The official Metadata Access Interface for EGEE

1
E-science grid facility for Europe and Latin
America
E-science grid facility for Europe and Latin
America
The AMGA Metadata Catalogue
Riccardo Bruno riccardo.bruno_at_ct.infn.it INFN
Catania, EELA-2 NA2 Training Manager 1st EELA-2
Grid School (E2GRIS1), 02nd -15th Nov 2008
www.eu-eela.org
2
Contents

• Metadata services background and possible uses on
  a grid environment
• Architecture and features of the gLite Metadata
  Service
• New AMGA Features
• existing DB import
• native SQL support
• Use cases

www.eu-eela.eu
www.eu-eela.eu
2
Itacuruçá (Brazil) , 1st EELA-2 Grid School
(E2GRIS1), 02.11.2008 15.11.2008
3
Why Grid needs Metadata?

• Grids allow to save millions of files spread over
  several storage sites.
• Users and applications need an efficient
  mechanism
• to describe files
• to locate files based on their contents
• This is achieved by
• associating descriptive attributes to files
• Metadata is data about data
• answering user queries against the associated
  information

www.eu-eela.eu
www.eu-eela.eu
3
Itacuruçá (Brazil) , 1st EELA-2 Grid School
(E2GRIS1), 02.11.2008 15.11.2008
4
Basic Metadata Concept

• Entries Representation of real world entities
  which we are attaching metadata to for describing
  them
• Attribute key/value pair
• Type The type (int, float, string,)
• Name/Key The name of the attribute
• Value - Value of an entry's attribute
• Schema A set of attributes
• Collection A set of entries associated with a
  schema
• Metadata - List of attributes (including their
  values) associated with entries

www.eu-eela.eu
www.eu-eela.eu
4
Itacuruçá (Brazil) , 1st EELA-2 Grid School
(E2GRIS1), 02.11.2008 15.11.2008
5
Example Movie Trailers

• Movie trailers files (entries) saved on Grid
  Storage Elements and registered into File
  Catalogue
• We want to add metadata to describe movie
  content.
• A possible schema
• Title -- varchar
• Runtime -- int
• Cast -- varchar
• LFN -- varchar
• A metadata catalogue will be the repository of
  the movies metadata and will allow to find
  movies satisfying users queries

www.eu-eela.eu
www.eu-eela.eu
5
Itacuruçá (Brazil) , 1st EELA-2 Grid School
(E2GRIS1), 02.11.2008 15.11.2008
6
Trailers example
www.eu-eela.eu
www.eu-eela.eu
6
Itacuruçá (Brazil) , 1st EELA-2 Grid School
(E2GRIS1), 02.11.2008 15.11.2008
7
Metadata service on the Grid

• Information about files -- but not only!
• metadata can describe any grid entity/object
• ex JobIDs - add logging information to your jobs
• monitoring of running applications
• ex ongoing results from running jobs can be
  published on the metadata server
• Inputset for a storm of parametric jobs
• information exchanging among grid peers
• ex producers/consumers job collections master
  jobs produce data to be analyzed slave jobs
  query the metadata server to retrieve input to
  consume
• Simplified DB access on the grid
• Grid applications that needs structured data can
  model their data schemas as metadata

www.eu-eela.eu
www.eu-eela.eu
7
Itacuruçá (Brazil) , 1st EELA-2 Grid School
(E2GRIS1), 02.11.2008 15.11.2008
8
Inputset for parametric jobs

• /grid/my_simulation/input
• --------------------------------------------------
  --------------------------------------------------
  
• entry     x1        x2        y1        y2  
       step      isTaken   found     output  
   
• -------------------------------------------------
  -------------------------------------------------
  
• 1         9453.1    9453.32   -439.93  
  -439.91   0.0006    JobID1234 No pillars    
       
• 2         9342.13   3435      3423    
   2343.2    0.003     No                    
         
• 3 34254.3 342342 432.43 132
  0.002 No
  
• ...... and so on
  
  
• --------------------------------------------------
  --------------------------------------------------
  
• This collection lists all the parameter set to be
  run on the Grid
• On the WN, one of the inputset is selected and
  isTaken is set JOB_ID of the job that has
  fetched it
• Results is also written in the found column to
  monitor the simulation
• so users can check the simulation from a UI,
  querying the metadata server, or from a WebPage
  (using APIs for ex)
• StdOutput can be copied also into the output
  text column

www.eu-eela.eu
www.eu-eela.eu
8
Itacuruçá (Brazil) , 1st EELA-2 Grid School
(E2GRIS1), 02.11.2008 15.11.2008
9
A possible parameter-get.sh script

• !/bin/bash
• Find the first set of parameters that has not
  been taken by noone
• IDmdcli find /grid/my_simulation/input
  'isTaken"No"' head -1
• Exit if all the parameters set has been already
  analyzed
• if "ID" "" then exit 1 fi
• set isTaken as its JOB_ID so that no one else
  will analyze the same set of parameter
• mdcli setattr /grid/my_simulation/input/ID
  isTaken echo GLITE_WMS_JOBID
• retrieve the set of the parameter to be scanned
• X1mdcli getattr /grid/my_simulation/input/ID
  x1 tail -1
• Y1mdcli getattr /grid/my_simulation/input/ID
  y1 tail -1
• X2mdcli getattr /grid/my_simulation/input/ID
  x2 tail -1
• Y2mdcli getattr /grid/my_simulation/input/ID
  y2 tail -1
• STEPmdcli getattr /grid/my_simulation/input/ID
  step tail -1
• Run the scan with the proper parameter and save
  the output to output.txt
• java -cp issgc_sfk_nesc.jarsfkscanner.jar
   uk.ac.nesc.toe.sfk.radar.Scanner X1 Y1 X2 Y2
  STEP gt output.txt
• the Scanner class returns the writing "No
  pillars found in this area" or "Found area" - so
  this will give useful info for monitoring during
  the run
• mdcli setattr /grid/my_simulation/input/ID found
  cat output.txt grep -i found

www.eu-eela.eu
www.eu-eela.eu
9
Itacuruçá (Brazil) , 1st EELA-2 Grid School
(E2GRIS1), 02.11.2008 15.11.2008
10
Monitoring of running application
showing results as long as they are produced
SE
/results collection
Workload Manager
Customer/Scientist
Scientist/Developer submitting jobs
www.eu-eela.eu
www.eu-eela.eu
10
Itacuruçá (Brazil) , 1st EELA-2 Grid School
(E2GRIS1), 02.11.2008 15.11.2008
11
Use a Metadata services to exchange data among
running jobs

• Suppose we have two sets of jobs
• Producers they generate a file, store on a SE,
  register it onto the LFC File Catalogue assigning
  a LFN
• Consumers they will take a LFN, download the
  file and elaborate it
• A Metadata collection can be used to share the
  information generated by the Producers it could
  act as a bag-of-LFNs (bag-of-task model) from
  which Consumers can fetch file for further
  elaboration

www.eu-eela.eu
www.eu-eela.eu
11
Itacuruçá (Brazil) , 1st EELA-2 Grid School
(E2GRIS1), 02.11.2008 15.11.2008
12
Information exchanging among grid peers
SE
Producers jobs
Consumers jobs
put LFN
fetch LFN
/bag-of-LFNs collection
Workload Manager
Scientist/Developer submitting jobs
www.eu-eela.eu
www.eu-eela.eu
12
Itacuruçá (Brazil) , 1st EELA-2 Grid School
(E2GRIS1), 02.11.2008 15.11.2008
13
The AMGA Metadata Catalogue

• Official metadata service for the gLite
  middleware
• but no dependencies from gLite software
• it can be used with other grid technologies/other
  environments
• AMGA Arda Metadata Grid Application
• Provide a complete but simple interface, in order
  to make all users able to use it easily.
• Designed with scalability in mind in order to
  deal with large number of entries
• based on a lightweight and streamed text-based
  protocol, like HTTP/SMTP
• Grid security is provided to grant different
  access levels to different users.
• Flexible with support to dynamic schemas in order
  to serve several application domains
• Simple installation by tar source, RPMs or
  Yum/YAIM

www.eu-eela.eu
www.eu-eela.eu
13
Itacuruçá (Brazil) , 1st EELA-2 Grid School
(E2GRIS1), 02.11.2008 15.11.2008
14
AMGA Analogies

• Analogy to the RDBMS world
• schema ?? table schema
• collection ?? db table
• attribute ?? schema column
• entry ?? table row/record
• Analogy to file system
• Collection ?? Directory
• Entry ?? File
• Example
• createdir /jobs (create table jobs)
• addattr /jobs jobStatus int (alter table jobs add
  column jobStatus int)
• addentry /jobs/job1 jobStatus 0 (insert into jobs
  (jobstatus) values(1))
• updateattr /jobs jobStatus 1 jobIDgt100 (update
  jobs set jobStatus1 where JobIDgt100)

www.eu-eela.eu
www.eu-eela.eu
14
Itacuruçá (Brazil) , 1st EELA-2 Grid School
(E2GRIS1), 02.11.2008 15.11.2008
15
AMGA Features

• Dynamic Schemas
• Schemas can be modified at runtime by client
• Create, delete schemas
• Add, remove attributes
• AMGA collections are hierarchical organized
• Collections can contain sub-collections
• Sub-collections can inherit/extend parent
  collection schema
• Flexible Queries
• SQL-like query language
• Different join type (inner, outer, left, right)
  between schemas are provided

selectattr /gLibraryFileName /gLAudioAuthor
/gLAudioAlbum '/gLibraryFILE/gLAudioFILE
and like(/gLibraryFileName, .mp3")

• Support for Views, Constraints, Indexes

www.eu-eela.eu
www.eu-eela.eu
15
Itacuruçá (Brazil) , 1st EELA-2 Grid School
(E2GRIS1), 02.11.2008 15.11.2008
16
Example
www.eu-eela.eu
www.eu-eela.eu
16
Itacuruçá (Brazil) , 1st EELA-2 Grid School
(E2GRIS1), 02.11.2008 15.11.2008
17
AMGA Security

• Unix style permissions - users and groups
• ACLs Per-collection or per-entry (table row).
• Secure client/server connections SSL
• Client Authentication based on
• Username/password
• General X509 certificates (DN based)
• Grid-proxy certificates (DN based)
• VOMS support
• VO attribute maps to defined AMGA user
• VOMS Role maps to defined AMGA user
• VOMS Group maps to defined AMGA group

www.eu-eela.eu
www.eu-eela.eu
17
Itacuruçá (Brazil) , 1st EELA-2 Grid School
(E2GRIS1), 02.11.2008 15.11.2008
18
AMGA Implementation

• C multiprocess server
• Backends
• Oracle, MySQL 4/5, PostgreSQL, SQLite
• Front Ends
• TCP text streaming
• High performance
• Client API for C, Java, Python, Perl, PHP
• SOAP (deprecated)
• Interoperability
• Scalability
• WS-DAIR Interface (new in AMGA 2.0)
• WS-enable environment
• Standalone Python Library implementation
• Data stored on file system

• AMGA server runs on SLC3/4, Fedora Core, Gentoo,
  Debian

www.eu-eela.eu
www.eu-eela.eu
18
Itacuruçá (Brazil) , 1st EELA-2 Grid School
(E2GRIS1), 02.11.2008 15.11.2008
19
AMGA Datatypes

• Using the above datatypes you are sure that your
  metadata can be easily moved to all supported
  back-ends
• If you do not care about DB portability, you can
  use, in principle, as entry attribute type ALL
  the datatypes supported by the back-end, even the
  more esoteric ones (PostgreSQL Network Address
  type or Geometric ones)

www.eu-eela.eu
www.eu-eela.eu
19
Itacuruçá (Brazil) , 1st EELA-2 Grid School
(E2GRIS1), 02.11.2008 15.11.2008
20
Performance and statistics
www.eu-eela.eu
www.eu-eela.eu
20
Itacuruçá (Brazil) , 1st EELA-2 Grid School
(E2GRIS1), 02.11.2008 15.11.2008
21
Accessing AMGA from UI/WNs

• TCP Streaming Front-end
• mdcli mdclient CLI and C API (md_cli.h,
  MD_Client.h)
• Java Client API and command line mdjavaclient.sh
  mdjavacli.sh (also under Windows !!)
• Python and Perl Client API
• PHP Client API NEW
• developed totally by the GILDA team INFN CT
• AMGA Web Interface (AMGA WI) ---NEW
• Developed totally by the GILDA team INFN CT
• Based on JAVA AMGA Standard APIs
• Web Application using standard as JSP Custom
  Tags, Servlet
• SOAP Frontend (WSDL)
• C gSOAP
• AXIS (Java)
• ZSI (Python)

www.eu-eela.eu
www.eu-eela.eu
21
Itacuruçá (Brazil) , 1st EELA-2 Grid School
(E2GRIS1), 02.11.2008 15.11.2008
22
AMGA Web Interface
www.eu-eela.eu
www.eu-eela.eu
22
Itacuruçá (Brazil) , 1st EELA-2 Grid School
(E2GRIS1), 02.11.2008 15.11.2008
23
Collection Management
www.eu-eela.eu
www.eu-eela.eu
23
Itacuruçá (Brazil) , 1st EELA-2 Grid School
(E2GRIS1), 02.11.2008 15.11.2008
24
Advanced features Metadata Replication

• AMGA provides a replication/federation mechanisms
• Motivation
• Scalability Support hundreds/thousands of
  concurrent users
• Geographical distribution Hide network latency
• Reliability No single point of failure
• DB Independent replication Heterogeneous DB
  systems
• Disconnected computing Off-line access
  (laptops)
• Architecture
• Asynchronous replication
• Master-slave writes only allowed on the master
• Application level replication
• Replicate Metadata commands
• Partial replication supports replication of
  only sub-trees of the metadata hierarchy

www.eu-eela.eu
www.eu-eela.eu
24
Itacuruçá (Brazil) , 1st EELA-2 Grid School
(E2GRIS1), 02.11.2008 15.11.2008
25
Metadata Replication Use cases
Partial replication
Full replication
Federation
Proxy
www.eu-eela.eu
www.eu-eela.eu
25
Itacuruçá (Brazil) , 1st EELA-2 Grid School
(E2GRIS1), 02.11.2008 15.11.2008
26
Existing DB access with AMGA

• Since AMGA 1.2.10, a new import feature allow to
  access existing DB tables
• Once imported into AMGA the tables from one or
  more DBs you want to access through AMGA, you can
  exploit many of the features brought to you by
  AMGA for your existing tables
• Advantages
• your db tables can be accessed by grid
  users/applications, using grid authentication
  (VOMS proxies)/authorization with ACLs
• exploiting AMGA federation features you can
  access several databases together from the Grid

www.eu-eela.eu
www.eu-eela.eu
26
Itacuruçá (Brazil) , 1st EELA-2 Grid School
(E2GRIS1), 02.11.2008 15.11.2008
27
Set up AMGA to access your tables

• To remember AMGA stores its own tables in its DB
  backend
• To access and existing DB you have 2 option
• import the tables of the DB you want to access to
  into AMGA DB backend
• viceversa, add AMGA DB backed tables to the DB
  you want to access to
• Use the import command by root to mount your
  table into the AMGA collection hierarchy
• Querygt whoami
• gtgt root
• Querygt createdir /world
• Querygt cd /world/
• Querygt import world.City /world/City
• Querygt import world.Country /world/Country
• Querygt import world.CountryLanguage
  /world/CountryLanguage

www.eu-eela.eu
www.eu-eela.eu
27
Itacuruçá (Brazil) , 1st EELA-2 Grid School
(E2GRIS1), 02.11.2008 15.11.2008
28
Set up AMGA to access your tables

• Properly set up authorization on the imported
  tables
• Querygt acl_remove /world/City/ systemanyuser
• Querygt acl_remove /world/Country systemanyuser
• Querygt acl_add /world/ gildausers rx
• Querygt acl_show /world
• gtgt root rwx
• gtgt gildausers rx
• gtgt systemanyuser rx
• Querygt selectattr CityCountryCode CityName
  'like(CityName, "Am") limit 5'
• gtgt NLD
• gtgt Amsterdam
• gtgt NLD
• gtgt Amersfoort
• gtgt BRA
• gtgt Americana
• gtgt ECU
• gtgt Ambato
• gtgt IDN
• More information on existing DB access _at_

www.eu-eela.eu
www.eu-eela.eu
28
Itacuruçá (Brazil) , 1st EELA-2 Grid School
(E2GRIS1), 02.11.2008 15.11.2008
29
DB Access and Replication
www.eu-eela.eu
www.eu-eela.eu
29
Itacuruçá (Brazil) , 1st EELA-2 Grid School
(E2GRIS1), 02.11.2008 15.11.2008
30
Native SQL Support

• Objective
• implement native SQL query processing
  functionality in AMGA
• Current Status
• direct SQL data statement in SQL92 Entry Level
  has been implemented in the 1.9 release
• Including 4 statements SELECT, DELETE, UPDATE
  and INSERT
• ALL SQL commands should be issued in UPPERCASE
• Entry name
• when a new entry is created with
  addentry/addentries, a name has to be assigned
  (filling the file column in the AMGA db
  backend)
• in the INSERT implementation, its filled
  automatically with a random guid

www.eu-eela.eu
www.eu-eela.eu
30
Itacuruçá (Brazil) , 1st EELA-2 Grid School
(E2GRIS1), 02.11.2008 15.11.2008
31
Native SQL Support

• Permission handling
• grant/revoke statemant are not supported
• ACL can be changed using the existing AMGA
  commands
• DB entity mapping
• DB Table Name AMGA Directory/Collection
• DB TableName.attribute AMGA TableNameattribute
• Testing
• PostgreSQL backend
• Plain table, permission, view, schema have not
  fully tested
• final version into AMGA 2.0 after summer and
  presented officially at the EGEE conference in
  Istanbul

www.eu-eela.eu
www.eu-eela.eu
31
Itacuruçá (Brazil) , 1st EELA-2 Grid School
(E2GRIS1), 02.11.2008 15.11.2008
32
Native SQL example

• Querygt INSERT INTO City VALUES
  (1,'Kabul','AFG','Kabol',1780000)
• gtgt Operation Success
• Querygt dir /world/City/
• gtgt /world/City/80b4fe646ed11dda02100304873049
• gtgt entry
• Querygt SELECT COUNT () FROM /world/City
• gtgt 3429
• Querygt SELECT FROM /world/City WHERE Name LIKE
  'Catani'
• gtgt 1472
• gtgt Catania
• gtgt ITA
• gtgt Sisilia
• gtgt 337862
• Querygt SELECT /world/CityName,
  /world/CityDistrict, /world/CountryName,
  /world/CountryRegion, /world/CountryContinent
  FROM /world/City, /world/Country WHERE
  /world/CityName LIKE 'Catani' AND Code 'ITA'
• gtgt Catania
• gtgt Sisilia
• gtgt Italy
• gtgt Southern Europe
• gtgt Europe

www.eu-eela.eu
www.eu-eela.eu
32
Itacuruçá (Brazil) , 1st EELA-2 Grid School
(E2GRIS1), 02.11.2008 15.11.2008
33
Early adopters of AMGA

• LHCb-bookkeeping
• Migrated bookkeeping metadata to ARDA prototype
• 20M entries, 15 GB
• Large amount of static metadata
• Feedback valuable in improving interface and
  fixing bugs
• AMGA showing good scalability
• Ganga
• Job management system
• Developed jointly by Atlas and LHCb
• Uses AMGA for storing information about job
  status
• Small amount of highly dynamic metadata

www.eu-eela.eu
www.eu-eela.eu
33
Dublin (Ireland) , Tutorial for User and Site
Admin., 15.09.2008 18.09.2008
34
Biomed - MDM

• Medical Data Manager MDM
• Store and access medical images and associated
  metadata on the Grid
• Built on top of gLite 1.5 data management system
• Demonstrated at last EGEE conference (October 05,
  Pisa)
• Strong security requirements
• Patient data is sensitive
• Data must be encrypted
• Metadata access must be restricted to authorized
  users
• AMGA used as metadata server
• Demonstrates authentication and encrypted access
• Used as a simplified DB
• More details at
• https//uimon.cern.ch/twiki/bin/view/EGEE/DMEncryp
  tedStorage

www.eu-eela.eu
www.eu-eela.eu
34
Itacuruçá (Brazil) , 1st EELA-2 Grid School
(E2GRIS1), 02.11.2008 15.11.2008
35
gMOD grid Movie On Demand

• gMOD provides a Video-On-Demand service
• User chooses among a list of video and the chosen
  one is streamed in real time to the video client
  of the users workstation
• For each movie a lot of details (Title, Runtime,
  Country, Release Date, Genre, Director, Case,
  Plot Outline) are stored and users can search a
  particular movie querying on one or more
  attributes
• Two kind of users can interact with gMOD
  TrailersManagers that can administer the db of
  movies (uploading new ones and attaching metadata
  to them) GILDA VO users (guest) can browse,
  search and choose a movie to be streamed.

www.eu-eela.eu
www.eu-eela.eu
35
Itacuruçá (Brazil) , 1st EELA-2 Grid School
(E2GRIS1), 02.11.2008 15.11.2008
36
gMOD under the hood

• Built on top of gLite services
• Storage Elements, sited in different place,
  physically contain the movie files
• LFC, the File Catalogue, keeps track in which
  Storage Element a particular movie is located
• AMGA is the repository of the detailed
  information for each movie, and makes possible
  queries on them
• The Virtual Organization Membership Service
  (VOMS) is used to assign the right role to the
  different users
• The Workload Management System (WMS) is
  responsible to retrieve the chosen movie from the
  right Storage Element and stream it over the
  network down to the users desktop or laptop

www.eu-eela.eu
www.eu-eela.eu
36
Itacuruçá (Brazil) , 1st EELA-2 Grid School
(E2GRIS1), 02.11.2008 15.11.2008
37
gMOD interactions
Storage Elements
VOMS
GENIUS Portal
AMGA
get Role
Workload Management System
User
www.eu-eela.eu
www.eu-eela.eu
37
Itacuruçá (Brazil) , 1st EELA-2 Grid School
(E2GRIS1), 02.11.2008 15.11.2008
38
gMOD screenshot
gMOD is accesible through the Genius Portal
(https//glite-demo.ct.infn.it)
www.eu-eela.eu
www.eu-eela.eu
38
Itacuruçá (Brazil) , 1st EELA-2 Grid School
(E2GRIS1), 02.11.2008 15.11.2008
39
GSAF Framework(Grid Storage Access Framework)

• GSAF is an Object Oriented Framework
• built on top of the Grid Metadata Service and
  Grid Data Service and exposes classes and related
  methods for applications

• Main objective
• hide the complexity and the fragmentation of the
  several underlying APIs
• satisfy functional requirements shared among
  applications
• ensure atomicity among different data
  manipulation operations

www.eu-eela.eu
www.eu-eela.eu
39
Dublin (Ireland) , Tutorial for User and Site
Admin., 15.09.2008 18.09.2008
40
GSAF System Architecture
The core of the framework is designed as a set of
plug-ins Its design covers several Object
Oriented Design Patterns (Singleton, Strategy
method, Factory method, Template Method,
Iterator and Composite). This ensures a very
clean and simple software architecture with an
high degree of cohesion and decoupling. Built on
top of Data Management Services of the Grid
Middleware
https//grid.ct.infn.it/twiki/bin/view/PI2S2/GSAF
www.eu-eela.eu
www.eu-eela.eu
40
Dublin (Ireland) , Tutorial for User and Site
Admin., 15.09.2008 18.09.2008
41
Class Diagram
www.eu-eela.eu
www.eu-eela.eu
41
Dublin (Ireland) , Tutorial for User and Site
Admin., 15.09.2008 18.09.2008
42
www.eu-eela.eu
www.eu-eela.eu
42
Dublin (Ireland) , Tutorial for User and Site
Admin., 15.09.2008 18.09.2008
43
www.eu-eela.eu
www.eu-eela.eu
43
Dublin (Ireland) , Tutorial for User and Site
Admin., 15.09.2008 18.09.2008
44
www.eu-eela.eu
www.eu-eela.eu
44
Dublin (Ireland) , Tutorial for User and Site
Admin., 15.09.2008 18.09.2008
45
ADAT project
www.eu-eela.eu
www.eu-eela.eu
45
Dublin (Ireland) , Tutorial for User and Site
Admin., 15.09.2008 18.09.2008
46
What is gLibrary

• gLibrary challenge is to offer a multiplatform,
  flexible, secure and intuitive system to handle
  digital assets on a Grid Infrastructure.
• By Digital Asset, we mean any kind of content
  and/or media represented as a computer file.
  Examples
• Images
• Videos
• Presentations
• Office documents
• E-mails, web pages
• Newsletters, brochures, bulletins, sheets,
  templates
• Receipts, e-books
• ... (only the imagination can make a limit)
• It allows to store, organize, search and retrieve
  those assets on a Grid environment.

www.eu-eela.eu
www.eu-eela.eu
46
Itacuruçá (Brazil) , 1st EELA-2 Grid School
(E2GRIS1), 02.11.2008 15.11.2008
47
Store assets on the Grid

• Users local assets are uploaded to one or more
  (as replicas) Storage Systems the user is
  authorized on
• Uploads are managed through Java Applets a
  direct GSIFTP copy is done from the local file to
  the chosen Storage Element

• File already on the Grid can be managed by
  gLibrary too
• a File Catalogue browser is integrated to select
  existing grid files.

www.eu-eela.eu
www.eu-eela.eu
47
Dublin (Ireland) , Tutorial for User and Site
Admin., 15.09.2008 18.09.2008
48
Organize assets

• All entries are organized according to their
  type
• a list of specific attributes to describe each
  kind of assets to be managed by the system
• hierarchical (child type shares parents
  attributes)
• defined by the gLibrary administrators
• queried by users

• Assets can be organized also by category
• Group together related assets of different types
• Useful also to define subset of assets belonging
  to the same type
• Multiple category assignment per asset

www.eu-eela.eu
www.eu-eela.eu
48
Dublin (Ireland) , Tutorial for User and Site
Admin., 15.09.2008 18.09.2008
49
Search assets

• Assets are browsed selecting a type (or category)
  and selecting one or more filters
• type attributes chosen from a defined list, used
  to narrow the result set
• Filter application is cascading and
  context-sensitive the selection of a filter
  value dynamically influences subsequent filter
  values (à la iTunes browser)
• Classic search available too

www.eu-eela.eu
www.eu-eela.eu
49
Dublin (Ireland) , Tutorial for User and Site
Admin., 15.09.2008 18.09.2008
50
Retrieve assets from the Grid

• User is presented with a list of asset replicas
• Download from the chosen storage element is
  matter of a mouse click
• Transfer handled over GridFTP with a Java Applet

www.eu-eela.eu
www.eu-eela.eu
50
Dublin (Ireland) , Tutorial for User and Site
Admin., 15.09.2008 18.09.2008
51
Features

• Implemented as Web 2.0 application
• AJAX and Javascript are strongly used to offer a
  desktop like user experience
• Business logic implemented using PHP 5 OOP support

www.eu-eela.eu
www.eu-eela.eu
51
Dublin (Ireland) , Tutorial for User and Site
Admin., 15.09.2008 18.09.2008
52
Browsing screenshot
www.eu-eela.eu
www.eu-eela.eu
52
Dublin (Ireland) , Tutorial for User and Site
Admin., 15.09.2008 18.09.2008
53
Entry detail screenshot
www.eu-eela.eu
www.eu-eela.eu
53
Dublin (Ireland) , Tutorial for User and Site
Admin., 15.09.2008 18.09.2008
54
Upload screenshot
www.eu-eela.eu
www.eu-eela.eu
54
Dublin (Ireland) , Tutorial for User and Site
Admin., 15.09.2008 18.09.2008
55
gLibrary Architecture overview
VOMS Server
3. get role
4. find the right asset
Login applet
2. proxy transfer over HTTPS
1. local proxy creation
5. proxy retrieved over HTTPS
6. direct transfer from SE
User
Upload/Download applet
www.eu-eela.eu
www.eu-eela.eu
55
Itacuruçá (Brazil) , 1st EELA-2 Grid School
(E2GRIS1), 02.11.2008 15.11.2008
56
gLibraryfor a mammograms repository
www.eu-eela.eu
www.eu-eela.eu
56
Itacuruçá (Brazil) , 1st EELA-2 Grid School
(E2GRIS1), 02.11.2008 15.11.2008
57
gLibrary for De Roberto works
www.eu-eela.eu
www.eu-eela.eu
57
Dublin (Ireland) , Tutorial for User and Site
Admin., 15.09.2008 18.09.2008
58
De Robertos manuscripts
www.eu-eela.eu
www.eu-eela.eu
58
Dublin (Ireland) , Tutorial for User and Site
Admin., 15.09.2008 18.09.2008
59
De Roberto rep Upload
www.eu-eela.eu
www.eu-eela.eu
59
Dublin (Ireland) , Tutorial for User and Site
Admin., 15.09.2008 18.09.2008
60
Conclusion

• AMGA Metadata Service of gLite
• Part of gLite 3.1
• can be used with other mws
• Useful to realize simple Relational Schemas
• Integrated on the Grid Environment (Security)
• Replication/Federation features
• Importing existing databases and soon native SQL
  support
• Tests show good performance/scalability
• gLibrary AMGA based DL platform

www.eu-eela.eu
www.eu-eela.eu
60
Itacuruçá (Brazil) , 1st EELA-2 Grid School
(E2GRIS1), 02.11.2008 15.11.2008
61
References

• AMGA Web Site
• http//cern.ch/amga
• AMGA Manual
• http//amga.web.cern.ch/amga/downloads/amga-manual
  _1_3_0.pdf
• AMGA API Javadoc
• http//amga.web.cern.ch/amga/javadoc/index.html
• AMGA Web Frontend
• http//gilda-forge.ct.infn.it/projects/amgawi/
• AMGA Basic Tutorial
• https//grid.ct.infn.it/twiki/bin/view/GILDA/AMGAH
  andsOn
• More information on existing DB access _at_
• http//amga.web.cern.ch/amga/importing.html
• https//grid.ct.infn.it/twiki/bin/view/GILDA/AMGAD
  Baccess

www.eu-eela.eu
www.eu-eela.eu
61
Itacuruçá (Brazil) , 1st EELA-2 Grid School
(E2GRIS1), 02.11.2008 15.11.2008
62
gLibrary References

• gLibray BETA homepage
• https//glibrary.ct.infn.it
• gLibrary paper
• https//glibrary.ct.infn.it/glibrary/downloads/gLi
  brary_paper_v2.pdf

www.eu-eela.eu
www.eu-eela.eu
62
Itacuruçá (Brazil) , 1st EELA-2 Grid School
(E2GRIS1), 02.11.2008 15.11.2008
63
Questions?
www.eu-eela.eu
www.eu-eela.eu
63
Itacuruçá (Brazil) , 1st EELA-2 Grid School
(E2GRIS1), 02.11.2008 15.11.2008