An Application for the Data Grid : A Prototype of the Japanese Virtual Observatory System

1
An Application for the Data Grid A Prototype
of the Japanese Virtual Observatory System

• Masatoshi Ohishi
• National Astronomical Observatory of Japan
• and Sokendai
• masatoshi.ohishi_at_nao.ac.jp

2
Collaborators

• NAOJ
• Y. Mizumoto
• N. Yasuda
• Y. Shirasaki
• M. Tanaka
• Ochanomizu U.
• Y. Masunaga

• Fujitsu Ltd.
• K. Monzen
• K. Kawarai
• Y. Ishihara
• H. Yanaka
• Y. Yamaguchi
• M. Ishida
• N. Saito

3
Supported by

• Research Grant from the MEXT
• (????? ???
• ??????
• ?????
• ?????)

• Research Grant from the Sokendai
• (????????? ????)

4
SUBARU telescope in Hawaii (optical IR)
5
Orion nebula In the IR
6
More than 5,000 Galaxies
7
45m Radio Telescope at Nobeyama
8
Observational DBs are common !
9
ALMA in Chile (sub-mm wave)
10
Time and Spectral DimensionsThe Multiwavelength
Crab Nebulae
Crab star 1053 AD
X-ray, optical, infrared, and radio views of
the nearby Crab Nebula, which is now in a state
of chaotic expansion after a supernova explosion
first sighted in 1054 A.D. by Chinese Astronomers.
Slide courtesy of Robert Brunner _at_ CalTech.
11
Unknowns of Astronomy

• Physics just after the Big Bang ?
• How Galaxies were formed ?
• Averaged density of the Universe
• etc.
• ? statistical analyses for homogeneous survey
  data are crucial

12
The Nature of Astronomers

• Higher sensitivity !
• More Data !
• Wider Area !
• Multi Wavelength Data !

13
Data Productivity

• Nobeyama Radio 1TB/yr
• Subaru_at_Hawaii 20TB/yr
• ALMA(planned) PB/yr
• Flood of excellent data (survey data)
• Digitized Archived

14
HPC High Speed NETWORK

• GRID Technology
• ? it is possible to connect and seamlessly use
  remote DBs and computing facilities
• ? Virtual Observatory

15
Digital Universe
Data Cube
Dec.
?/Freq.
Numerical Universe

• R.A.

???????????
Time
Real Universe
16
Telescope is behind me !
17
(No Transcript)
18
When Data is Not Found

• ? VO requests REAL OBSERVATIONS to REAL
  telescopes
• ? Real Observations
• ? Get data
• ? the data is registered into the VOs
• VO is a growing facility !

19
JVO consists of

• Seamless Access to Federated DBs
• Various Data Analysis tools
• Advanced Analysis such as Data Mining
• Federation with Real Telescopes
• ? crucial to control data quality in VO

20
JVO prototype Ver. 1

• Establish seamless access to federated DBs by
  using the Grid technology
• Define JVO Query Language, and implement its
  parser
• Test environment is in a closed subnet

21
Data Bases in the prototype

• Subaru SuprimeCAM
• Sloan Digital Sky Survey (SDSS)
• 2µm All Sky Survey (2MASS)

22
Web Broser
????? ????
Catalog Query Service
Commands callable from GT2
JVO Portal
Parse the JVO QL into each query sequence
Catalog DB
Data Archive Service
JVO Contoroller
skycat
Globus Toolokit V2
Data Management DB
Data
UDDI Server (FreeSoft)
Globus Toolkit
Service Registry
Data Analysis Services
23
JVO Query Language(1)Catalog Query
24
JVO Query Language(2)Image Query
25
Redhat Linux
2Mass DataBase
JVO Prototype System Architecture
Redhat Linux
JVO Service
Gb Ether
Solaris8 (Possible at Linux)
JVO Controller
GSDL
UDDI Maintenance Tool
UDDI Registory / Servlet Engine
Globus Toolkit (Server)
Java2SE 1.4
Java2SE 1.4
Apache Tomcat
JVO Service (X match)
Maintenance Tool
soapuddi
UDDI4J
SQL library
JVO Service (select)
Apache SOAP

• register
• update
• delete

JDBC
? Applet DownLoad
? Find Service
? Request Observation
JVO Controller
DB Service
DB Service
Image Data (copy)
Java2SE 1.4
PostgreSQL (DBMS)
PostgreSQL
JVO Server
Monitoring
2Mass DB
Image Data
UDDI4J
SUBARU SupCam z-band
UDDI Data

• get parse GSDL
• pre condition check
• execute Service
• post condition check

SUBARU DataBase
Redhat Linux
JVO Service
? issue SQL
? execute Service
? get GSDL
GSDL (copy)
GSDL
? Get GSDL by GridFTP
? execute Service
Check status
Globus Toolkit (Client)
Globus Toolkit (Server)
Internet
? spawn Service
JVO Service (image)
? Get GSDL by GridFTP
AVO
JVO Service
JVO Service (X match)
SUBARU DataBase
Globus Toolkit
Globus Toolkit (Server)
GridFTP
GSDL
SQL library
JVO Service (select)
NVO
? spawn Service
JVO Service (count)
JVO Service (select)
SQL library
SQL library
JVO Service (image)
JVO Service (X match)
Globus Toolkit
SQL library
JVO Service (count)
? issue SQL
Image Data (copy)
DB Service
Image Data
Copy imate data by GridFTP
Image Data (copy)
ORACLE
Copy imate data by GridFTP

• The following free software is included by globus
  toolkit.
• OpenSSL (Secure Socket Library)
• OpenLDAP (LDAP Server)
• wu-ftpd (ftp server)

SUBARU SupCam i-band
Image Data
Free Software
In House Software
Commercial Software
Function
26
How it works ?

• Read JVO QL
• Create Work Flow, and execute each step
• When necessary, access to the registry to expand
  the step of the work flow
• Thus the steps increases dynamically

27
JVO Prototype Architecture
Solaris8 (Possible at Linux)
Redhat Linux
JVO Service
UDDI Registory / Servlet Engine
UDDI Maintenance Tool
Java2SE 1.4
Java2SE 1.4
Globus Toolkit (Server)
Apache Tomcat 4.x
Maintenance Tool
SQL library
JVO Service
soapuddi
UDDI4J
Hypertext contents
Apache SOAP
JDBC
DB Service
Applet Down Load
JVO Client
PostgreSQL
RDB
JVO Server
Netscape Communicator
Java2 Plugin
Java2SE 1.4

JVO Client (applet)
JVO Controller
Redhat Linux
UDDI4J
JVO Service
Globus Toolkit (Client)
Globus Toolkit (Server)
DB Service
JVO Service
SQL library
JVO Service
PostgreSQL
Globus Toolkit (Server)
DB Service
JVO Service
SQL library
RDB
ORACLE
RDB

• The following free software is included by globus
  toolkit.
• OpenSSL (Secure Socket Library)
• OpenLDAP (LDAP Server)
• wu-ftpd (ftp server)

Free Software
In House Software
Commercial Software
Function
28
(No Transcript)
29
(No Transcript)
30
(No Transcript)
31

• create materialized view my_table as
• select i.POS_EQ_RA_MAIN,
• i.POS_EQ_DEC_MAIN,
• i.PHOT_SDSS_I,
• z.POS_EQ_RA_MAIN,
• z.POS_EQ_DEC_MAIN,
• z.PHOT_SDSS_Z,
• img_i.BOX(POINT(ra1, dec1), 0.005,
  0.005) as img_i,
• img_z.BOX(POINT(ra2, dec2), 0.005,
  0.005) as img_z
• from Subaru.SPcam.i i,
• Subaru.SPcam.z z,
• Subaru.SPcam.i img_i,
• Subaru.SPcam.z img_z
• where XMATCH(i, z)
• and BOX(POINT(201., 27.4), 0.10,
  0.10)
• and i.PHOT_SDSS_I - z.PHOT_SDSS_Z
  0.0
• and i.PHOT_SDSS_I
• and z.PHOT_SDSS_Z

32
(No Transcript)
33
(No Transcript)
34
(No Transcript)
35
(No Transcript)
36
VOTable as output

• XML
• Standard data exchange format among VOs defined
  by the IVOA (see below)

37
(No Transcript)
38
(No Transcript)
39
FAQs on our Prototype

• Q) Contents of environment variables are not
  passed to the executing machines.
• A) Prepare a shell script to get environment
  variables and another script to execute the
  command. Then GridFTP these shell scripts , and
  execute the shells in the remote machine.
• Q) It is impossible to prepare all users account
  to all relevant machines.
• A) Map all accounts on a portal to a single
  representative one, such as globus.

40
FAQs

• Q) GT2 has too large overhead before starting
  actual jobs !! They are several seconds.
• A) For batch jobs, such as database accesses, the
  overhead could be negligible. However users can
  never wait for several second for interactive
  data analyses.
• Request) Overheads must be
• excluding the network delays.

41
Future

• Prototype version 2
• implement data analysis tools
• better user interface
• Interoperation with VOs in the world
• We have joined the International Virtual
  Observatory Alliance, and are contributing
  standardizing protocols for interoperability.

42
VOs in the World

• NVO (USA), AVO (EU),AstroGrid (UK)
• Germany, Canada, Russia, Australia
• India, China, JAPAN
• Protocols to interconnect VOs
• ? International Virtual Observatory Alliance
  (IVOA)

43
www.ivoa.net
44
MVC
????
MVC
??????DB
??????
MVC
Web ???
JVO ????
VO? ????
?????
????
????????
?????? ????
???? ??
???
VO? Viewer
??? ???
????
DA/DB
DB
Data
???? ?????A
?????
????
????????
?????
????
????????
GRID?? A
???? ?????C
???? ?????B
Web ???
VO ????
VO? ????
????????
???
VO? Viewer
????

?????
?????
??
GRID?? C
GRID?? B
45
JVO Home Page

• http//jvo.nao.ac.jp/