Status and Prospective in Japan

1
Status and Prospective in Japan

• Atsushi Manabe
• Computing Research Center (CRC)
• KEK

2
Outline

• Grid projects (budget) in Japan academic region.
• HEP Data Grid activities in Japan.
• KEK
• Needs in KEK
• Strategy of KEK CRC
• Pacifi-Grid
• Gfarm project
• Grid/HPSS test bed on SupserSINET.
• ICEPP ATLAS Japan Regional Center

3
Grid Projects(budget) in Japan (academic area
only)

• MEXT(Ministry of Education, sports . and
  technology) projects
• SuperSINET Gigabit backbone for academic
  researchers in Japan
  
• MEXT Informatics Project
• A05 Application Grid projects for science
  US 6M /5 y
• Astronomy (Virtual Observatory),
  io-Infomatics/Genome (BioGrid)
• Networking supercomputers ITBL
  US 100M /5y
• National Research Grid Initiative
  US 300M/5y focused in Nano-tech
  application
• (LHC/ATLAS(ICEPP,KEK)
• METI (Ministry of Economy, Trade and Industry)
  project
• AIST GTRC(Grid Technology Research Center)
  US 140M

4
Gigabit network

• Super SINET
• 10 Gbps MEXT DWDM nationwide network
• About 40 universities and labs
• US 60M/yr
• 5 application areas HEP, Nanotech,
  Bioinformatics, Astronomy, Grid Computing
• Tsukuba-WAN
• 330Gbps DWDM ring in the Tsukuba area
• Tens of national labs
• 6 supercomputers in TOP100
• Testbed for Grid infrastructure and applications

5
IP 10 Gbps
WDM path
NIFS
IMS
IP router
Nagoya U
NIG
Nagoya Hub
Hokkaido
Tokyo Hub
Osaka Hub
KEK
Osaka U
Tsukuba
US,EU
Kyoto U
Korea
NII
Kyushu U
ISAS
U Tokyo
ICR Kyoto-U
Internet
TITEC
NAO
Waseda
IMS
U-Tokyo
Network Topology of SuperSINET
6
SuperSINET Backbone
KEK
Titech
ICEPP
7
Network Infrastructure
TEIN
NII- NY
TRANSPAC
Taiwan-US
Japan (NII) -NY 2.4G x 2 Jan. 2003 Japan US
622M x 2 (TRANSPAC) Korea US 45 M Korea
Japan 2.4G Jan. 2003 Korea Europe 2 M
(TEIN) China(IHEP) Japan(KEK) 128 kbps
(HEP) China US 10 M Taiwan Japan 155
M Taiwan US 622 M (Dec. 2002)
8
MEXT Grid Projects

• MEXT Informatics Project A05 Grid Area
• Lead by Shinji Shimozo(Osaka U.) Satoshi
  Matsuoka(Titech)
• 11 groups of universities and labs
• Research and development on application,
  middleware, and infrastructure
• Support the Virtual Observatory(astronomy) and
  the BIO-Grid(bio-Infomatics/Genome) projects
• Tokyo Institute of Technology Titech Campus
  Grid (US 2M)
• Lead by Satoshi Matsuoka(Titech)
• Total of 800 CPU PC clusters at 13 locations in a
  30km range connected by Super TITANET(1-4Gbps)
  backbone
• Pilot project for virtual parallel computer
  infrastructure for high end computational
  e-Science

9
ITBL

• ITBL(IT-based Laboratory)
• To Connect Super-Computers in Government Labs
  NAL, RIKEN, NIED, NIMS, JST, JAERI
• Project period 2001-2005 (3-stage project) with
  total of US 105M funding
• Software development project.
• Applications mechanical simulation,
  computational biology, material science,
  environment, earthquake engineering
• Step 1 Supercomputer centers of government lab
  are networked via SuperSINET
• Step 2 Virtual Research Environment
  Grid-enabling laboratory applications
• Step 3 Sharing information among researchers
  from widely distributed disciplines and
  institutions

10
NAREGI (National Research Grid Initiative)

• NII (The National Institute of Informatics)
  under MEXT project.
• Making Center of Grid Research Development
• Development of Grid Lower and Middleware(Grid
  scheduler,VM, RPC, MPI, GUI tools)
• Networking, Security and User management(Routing,
  robust TCP/IP protocol, Traffic management)
• Applications for Nano-science and
  technology.Joint with Institute for Molecular
  Science.
• http//www.grid.nii.ac.jp/

11
Grid Technology Research Center

• Part of AIST under METI (Ministry of Economy,
  Trade and Industry)
• GTRC (Grid Technology Research Center)
• Established in Jan. 2002
• Director Satoshi Sekiguchi
• US140M for building, equipment, RD and
  operations
• AIST super server PC cluster2004-2Q 11TFLOPS/
  AMD Operon x 2116CPUs
• To be central institute for Grid research in
  Japan
• Not just RD, but serve as central CA, ApGrids
  principal site
• Collaboration with institutions and other Grid
  projects of both domestic and international scale
• Organizing GRID Consortium Japan
• http//unit.aist.go.jp/grid/

12
Grid related Activity at KEK
13
Major projects in KEK

• KEK sponsors projects which use accelerators and
  also projects in related fields
• KEKB accelerator
• BELLE experiment
• Proton synchrotron
• nuclear and elementary particle physics
• medical
• Synchrotron Radiation Facility
• bio, material science and so on
• Also many theorists in related fields
• Support International HEP collaboration
• ATLAS Japan

14
KEK Future projects

• J-PARC
• will be built in Tokai div. of Japan Atomic
  Energy Research Institute (JAERI) where 50km
  north from KEK start the operation in JFY2005
• 50 GeV high-intensity proton synchrotron
• nuclear physics and elementary particle physics
• Super B factory
• x30 Luminosity(require x10 CPU power). Under
  discussion.
• Linear Collider
• discussion is still under way

15
Computing Facilities at KEK

• Computing Research Center offers group oriented
  computing services
• Exclusive computing resource for main groups
• Networking, storage and computing resource
• User group owns their own computer systems
• PC farms

16
Main computing facilities

• KEKCC computer system
• for experiments at PS, linear collider RD,
  theory, ATLAS and so on
• IBM SP3, 120TB HPSS storage
• KEKB computer system for Belle
• Sun WSs and PC farms. 500TB storage.
• KEKSC Super-computer system for mainly lattice
  QCD G.
• Hitachi SR-8000 100 nodes almost 1TFLOPTS

17
Our needs in Grid Era

• ATLAS Regional Center (Grid is premise)
• Being set up at ICEPP (Intl Center for Particle
  Physics, U. Tokyo)
• Still KEK have large portion of ATLAS Japan
  members and they will make data analysis using
  computing resource at KEK. We will support them.
• And
• BELLE international collaboration and require
  more and more CPU and storage.
• J-PARC joint project of other Inst. regionally
  separated.

18
BELLE

• BELLE is the major project at KEK
• They have many collaborators world wide
• They always needs additional CPU power because
  acc. luminosity improving and improving.
• They seek a solution of their problems related
  computing issue using GRID.
• We interviewed the BELLE computing coordinator
  and got the wish list shown in the following
  slides

19
Wish lists from BELLE

• What BELLE wants
• A way to integrate CPU powers and way to
  distribute data among collaborators Univ.s and
  Inst.s for their data production.
• Automatic way to submit analysis jobs in the
  distributed environment.
• once analysis codes have placed somewhere,
  process for all of experimental data and Monte
  Carlo should be done automatically and simply
  users gain the result
• Automatic Monte Carlo production
• Automatic version control for libraries,
  constants and others on all of the computer
  system in BELLE
• Automatic update on reconstructed data or Monte
  Carlo if libraries or constants are once updated

20
Wish lists from BELLE 2

• It is not easy to achieve their requests
  technically
• simplest solution will be hire more Post-Docs or
  find more grad-students, however
• We have decomposed their requests
• should be done in application level (BELLE)
• or
• can be solved by GRID

21
What they need related GRID

• An efficient way to distribute jobs and way to
  manage/share/distribute data for the jobs.
• effective data catalogue
• physical to logical mapping
• Seamless and unique file tree
• query
• file names, contents and others
• works on the Internet but secure
• efficient and easy to distribute data to remote
  sites
• easy to use
• web interface and GUI also

22
continued

• mass storage management system
• not only tapes but also large amount of disks
• robust and fast transfer
• easy integration with other component
• We already have them partially on LAN, but we
  need to expand them to the Internet. We hope GRID
  will solve them.

23
KEK CRC strategy towards GRID
24
Done or On going.

• Install GT2(Globus toolkit ver.2) and
  applications (LSF) to two CRC Systems (KEKCC
  KEKB) ) and will open to some limited users.
  (remote Job submission)
• Build temporal CA with single Virtual
  Organization(VO).
• Build a inter-institute wide Grid test bed on
  SuperSINET and make a performance test. (talk
  later)
• Trial use of SRB with Belle data in a test
  system.
• Start to touch LCG tools by ATLAS Japan people.
• watch technical trends carefully.

25
SRB

• SRB is available today and looks useful even in
  HEP ( http//www.npaci.edu/DICE/SRB/ )
• developed at San Diego Super Computer Center
• seamless solution to HPSS
• good for the KEKCC (central computer) users
• Data catalogue and replication
• SLAC is also testing

26
Until early 2004

• Deploy GT3, evaluate LCG tools and other
  something new technology.
• Build own VO at each sites and integrate them.
• File sharing by SRB (or ?) and seamless job
  submission between KEKCC and KEKB
• Pacifi-Grid test environment Pan-Pacific
  collaboration (see later)
• KNU (Korea), Univ. of Melbourne (Australia), SLAC
  and KEK will test SRB
• Start to develop higher level software
  (experiment side) on the top of Globus and SRB.
• Message passing with GSI authentication ?

27
2004

• KEKB and KEKCC (5yeas rental )system replacement
  (2005-4Q) .
• GRID will become concrete until this stage or die
• Final specification have to be decided
• Decision of middle ware (Grid or else?)
• CA
• Security
• human resource
• trained engineer on site

28
Other topics discussed internally

• CA
• nation wide, institution wide or else? KEK
  certify all Japanese HEP user?
• Rules for certification of users.
• PKI
• secure enough? Safety private key management (IC
  card?)
• management costs?
• seamless integration with local authentication
• Network
• How we can overcome latency?
• better connectivity to the Internet, routing and
  throughput
• middleware
• Globus, Avaki or something else
• SRB, LCG or other data replication solution
• Storage
• SAN
• StorageTank (IBM)
• security vs usability

29
Pacifi-GridProposal on Open Pacific Grid
collaboration

• Takashi Sasaki (KEK)

30
charter

• Give research institutions and IT vendors
  opportunity of testing Grid technologies under
  inter-state environment.
• Emphasis on Data Grid
• Open to everybody who wants to join
• Grid is a newly emerging technology and more
  experiences are necessary before deploying at the
  large scale production systems
• Test on high latency network
• robustness tests
• interoperability tests among sites and vendors
• security

31
Collaboration (under discussion)

• Korea
• U of Melbourne, Australia
• SLAC, USA
• JAPAN
• ICEPP (Univ. of Tokyo)
• KEK
• Some venders
• Under discussing with Fujitsu, IBM Japan and so
  on

32
Gfarm RD

• Development RDGrid middleware of Cluster File
  System with job execution.

33
RD of Gfarm (Grid data farm)

• Gfarm
• Development by CS scientist of AIST, Titech and
  KEK since 2000
• AIST Advanced Inst. for Science and Technology
• Architecture
• PC farm with large local disk/node as a GRID
  fabric
• Large data file is divided into fragments and
  stored in the disks
• Data I/O by parallel file system
• Affinity scheduling of process and storage for
  data residence

34
Data Access Locality

• Exploit the data access locality as much as
  possible

35
Gfarm
http//datafarm.apgrid.org/

• Grid Data Farm Beta 3 release is now
  available.
• Parallel I/O exploit the data access locality,
  store and access files by fragments
• Parallel Job program runs on thenodes where the
  file fragmentsreside owner computes
• Programs and file fragments,job history, data
  checksum,file replication are managed
  withMetaDB
• Fragments are replicated forbackup and load
  balancing
• User sees the fragmented files asa single Gfarm
  URL
• Provide system call hooks foropen(), close(),
  read(), write() etc

• AuthenticationGlobus GSI and/orShared Private
  Key

36
FADS/Goofy in ATLAS

• Used in sub-detector software developments and
  physics validations
• Detector modules also run in ATLAS mainstream
  framework (Athena)
• Testbed for ROOT I/O
• Bandwidth Challenge inSC2002
• Generated 106 fully simulatedhigss ? 4? events
  in 2 dayswith 400 CPU

37
Gfarm achievement

• File affinity scheduling provides parallel
  processing capability for both I/O-bound and
  CPU-intensive jobs.
• FADS/Goofy was successfully implemented on Gfarm
  architecture. In the process, ROOT I/O module
  works with Gfarm parallel file system using the
  system call hook, without a changes to the ROOT
  package.
• ROOT I/O files has been successfully replicated
  at 2.286 Gbps using the SC2002 Bandwidth
  Challenge testbed with 12 nodes 190Mbps/node.
• Please refer to CHEP03 reports by Y.Morita and
  O.Tatebe

38
Status of ATLAS Japan RC (ICEPP U. of Tokyo)

• RD is ongoing
• Test-buy of PC farm and storage
• 1U P3, P4 servers ( gt150nodes and more) , LTO
  (10TB) with CASTOR and gt20TB disk servers.
• Nordu GRID was tested first
• Data transfer tests between U. Tokyo and KEK was
  done
• ATLAS data challenges
• LCG 0 and 1

39
(No Transcript)
40
(No Transcript)
41
LCG-0 Deployment Status
These sites deployed the LCG-0 pilot system and
will be the first sites to deploy LCG-1
Site Scheduled Status
Tier 1 Tier 1
0 CERN 15/2/03 Done
1 CNAF 28/2/03 Done
2 RAL 28/2/03 Done
3 FNAL 30/3/03 Done
4 Taipei 15/4/03 Done
5 FZK 30/4/03 Done
6 IN2P3 7/5/03 In prep.
7 BNL 15/5/03 Done
8 Russia (Moscow) 21/5/03 In prep.
9 Tokyo 21/5/03 Done
Tier 2 Tier 2
10 Legnaro (INFN) After CNAF Done
42
LCG Resource Commitments 1Q04
  CPU (kSI2K) Disk TB Support FTE Tape TB
CERN 700 160 10.0 1000
Czech Republic 60 5 2.5 5
France 420 81 10.2 540
Germany 207 40 9.0 62
Holland 124 3 4.0 12
Italy 507 60 16.0 100
Japan 220 45 5.0 100
Poland 86 9 5.0 28
Russia 120 30 10.0 40
Taiwan 220 30 4.0 120
Spain 150 30 4.0 100
Sweden 179 40 2.0 40
Switzerland 26 5 2.0 40
UK 1656 226 17.3 295
USA 801 176 15.5 1741
Total 5600 1169 120.0 4223
43
(No Transcript)
44
Status of ATLAS Japan RC 2

• Budget requests have been submitted to the
  funding agency
• still no decision has been made
• and no decision will be made within this FY
• All of national universities and some national
  laboratories including KEK will be
  non-nationalized (not privatized, but will be a
  kind of independent governmental agencies)

45
Summary of KEK Grid activity

• KEK had a step forward for the Grid computing
• People at ATLAS Japan RC are working on LCG
• Realistic environment is necessary to test
  middleware and applications
• We have started the discussion among Pan-Pacific
  sites to collaborate each other
• We want to exchange knowledge and experiences

46

• End of my presentation
• Thank you