JVO?????(2) ????

1
JVO?????(2)????

• ????
• ????? ???
• masatoshi.ohishi_at_nao.ac.jp

2
Collaborators

• NAOJ
• Y. Mizumoto
• N. Yasuda
• Y. Shirasaki
• M. Tanaka
• Ochanomizu U.
• Y. Masunaga

• Fujitsu Ltd.
• K. Monzen
• K. Kawarai
• Y. Ishihara
• H. Yanaka
• Y. Yamaguchi
• M. Ishida
• N. Saito

3
Supported by

• Research Grant from the MEXT
• (????? ???
• ??????
• ?????
• ?????)

• Research Grant from the Sokendai
• (????????? ????)

4
Data Productivity

• Nobeyama Radio 1TB/yr
• Subaru_at_Hawaii 20TB/yr
• ALMA(planned) PB/yr
• Flood of excellent data (survey data)
• Digitized Archived

5
HPC High Speed NETWORK

• GRID Technology
• ? it is possible to connect and seamlessly use
  remote DBs and computing facilities for data
  analyses
• ? Virtual Observatory

6
Digital Universe
Data Cube
Dec.
?/Freq.
Numerical Universe

• R.A.

???????????
Time
Real Universe
7
(No Transcript)
8
When Data is Not Found

• ? VO requests REAL OBSERVATIONS to REAL
  telescopes
• ? Real Observations
• ? Get data
• ? the data is registered into the VOs
• VO is a growing facility !

9
JVO (will) consists of

• Seamless Access to Federated DBs
• Various Data Analysis tools
• Advanced Analysis such as Data Mining
• Federation with Real Telescopes
• ? crucial to control data quality in VO

10
JVO prototype Ver. 1

• Establish seamless access to federated DBs by
  using the Grid technology
• Define JVO Query Language, and implement its
  parser
• Test environment is in a closed subnet

11
Data Bases used in the prototype

• Subaru SuprimeCAM
• 2µm All Sky Survey (2MASS)

12
Web Broser
????? ????
Catalog Query Service
Commands callable from GT2
JVO Portal
Parse the JVO QL into each query sequence
Catalog DB
Data Archive Service
JVO Contoroller
skycat
Globus Toolokit V2
Data Management DB
Data
UDDI Server (FreeSoft)
Globus Toolkit
Service Registry
Data Analysis Services
13
JVO Query Language
???,??? ???(V27b)?
14
Redhat Linux
2Mass DataBase
JVO Prototype System Architecture
Redhat Linux
JVO Service
Gb Ether
Solaris8 (Possible at Linux)
JVO Controller
GSDL
UDDI Maintenance Tool
UDDI Registory / Servlet Engine
Globus Toolkit (Server)
Java2SE 1.4
Java2SE 1.4
Apache Tomcat
JVO Service (X match)
Maintenance Tool
soapuddi
UDDI4J
SQL library
JVO Service (select)
Apache SOAP

• register
• update
• delete

JDBC
? Servlet DownLoad
? Find Service
? Request Observation
JVO Controller
DB Service
DB Service
Image Data (copy)
Java2SE 1.4
PostgreSQL (DBMS)
PostgreSQL
JVO Server
Monitoring
2Mass DB
Image Data
UDDI4J
SUBARU SupCam z-band
UDDI Data

• get parse GSDL
• pre condition check
• execute Service
• post condition check

SUBARU DataBase
Redhat Linux
JVO Service
? issue SQL
? execute Service
? get GSDL
GSDL (copy)
GSDL
? Get GSDL by GridFTP
? execute Service
Check status
Globus Toolkit (Client)
Globus Toolkit (Server)
Internet
? spawn Service
JVO Service (image)
? Get GSDL by GridFTP
AVO
JVO Service
JVO Service (X match)
SUBARU DataBase
Globus Toolkit
Globus Toolkit (Server)
GridFTP
GSDL
SQL library
JVO Service (select)
NVO
? spawn Service
JVO Service (count)
JVO Service (select)
SQL library
SQL library
JVO Service (image)
JVO Service (X match)
Globus Toolkit
SQL library
JVO Service (count)
? issue SQL
Image Data (copy)
DB Service
Image Data
Copy imate data by GridFTP
Image Data (copy)
ORACLE
Copy imate data by GridFTP

• The following free software is included by globus
  toolkit.
• OpenSSL (Secure Socket Library)
• OpenLDAP (LDAP Server)
• wu-ftpd (ftp server)

SUBARU SupCam i-band
Image Data
Free Software
In House Software
Commercial Software
Function
15

• JVO???????GRID???????
• ??????(V30a)???
• ????????,user interface?????
• ??????(V28b)???
• ??????????????
• ??????(V29b)???

16
How it works ?

• Read JVO QL
• Create Work Flow, and execute each step
• When necessary, access to the registry to expand
  the step of the work flow
• Thus the steps increases dynamically

17
JVO Prototype Architecture
Solaris8 (Possible at Linux)
Redhat Linux
JVO Service
UDDI Registory / Servlet Engine
UDDI Maintenance Tool
Java2SE 1.4
Java2SE 1.4
Globus Toolkit (Server)
Apache Tomcat 4.x
Maintenance Tool
SQL library
JVO Service
soapuddi
UDDI4J
Hypertext contents
Apache SOAP
JDBC
DB Service
Applet Down Load
JVO Client
PostgreSQL
RDB
JVO Server
Netscape Communicator
Java2 Plugin
Java2SE 1.4

JVO Client (applet)
JVO Controller
Redhat Linux
UDDI4J
JVO Service
Globus Toolkit (Client)
Globus Toolkit (Server)
DB Service
JVO Service
SQL library
JVO Service
PostgreSQL
Globus Toolkit (Server)
DB Service
JVO Service
SQL library
RDB
ORACLE
RDB

• The following free software is included by globus
  toolkit.
• OpenSSL (Secure Socket Library)
• OpenLDAP (LDAP Server)
• wu-ftpd (ftp server)

Free Software
In House Software
Commercial Software
Function
18
(No Transcript)
19
(No Transcript)
20
Query Files can be stored in a local PC (WS)
21

• create materialized view my_table as
• select i.POS_EQ_RA_MAIN,
• i.POS_EQ_DEC_MAIN,
• i.PHOT_SDSS_I,
• z.POS_EQ_RA_MAIN,
• z.POS_EQ_DEC_MAIN,
• z.PHOT_SDSS_Z,
• img_i.BOX(POINT(ra1, dec1), 0.005,
  0.005) as img_i,
• img_z.BOX(POINT(ra2, dec2), 0.005,
  0.005) as img_z
• from Subaru.SPcam.i i,
• Subaru.SPcam.z z,
• Subaru.SPcam.i img_i,
• Subaru.SPcam.z img_z
• where XMATCH(i, z) lt 10 arcsec NEAREST
• and BOX(POINT(201., 27.4), 0.10,
  0.10)
• and i.PHOT_SDSS_I - z.PHOT_SDSS_Z gt
  0.0
• and i.PHOT_SDSS_I lt 20.0
• and z.PHOT_SDSS_Z lt 20.0

22
Monitor Window Refreshed automatically in every
15 seconds
23
VOTable as output

• XML
• Standard data exchange format among VOs defined
  by the IVOA (see below)

24
Sample Outputs
25
(No Transcript)
26
FAQs on our Prototype

• Q) Contents of environment variables are not
  passed to the executing machines.
• A) Prepare a shell script to get environment
  variables and another script to execute the
  command. Then GridFTP these shell scripts , and
  execute the shells in the remote machine.
• Q) It is impossible to prepare all users account
  to all relevant machines.
• A) Map all accounts on a portal to a single
  representative one, such as globus.

27
FAQs

• Q) GT2 has too large overhead before starting
  actual jobs !! They are several seconds.
• Request) Overheads must be lt 0.1 seconds
  excluding the network delays.
• A) Use NMI (NSF Middleware Initiative) where each
  key generation is quite fast through assembler.

28
Future

• Prototype version 2 (in progress)
• implement data analysis tools
• add more data (SXDF, SDSS, etc.)
• Interoperation with VOs in the world
• We have joined the International Virtual
  Observatory Alliance, and are contributing
  standardizing protocols for interoperability.

29
VOs in the World

• NVO (USA), AVO (EU),AstroGrid (UK)
• Germany, Canada, Russia, Australia
• India, China, JAPAN
• Protocols to interconnect VOs
• ? International Virtual Observatory Alliance
  (IVOA)

30
MVC
????
MVC
??????DB
??????
MVC
Web ???
JVO ????
VO? ????
?????
????
????????
?????? ????
???? ??
???
VO? Viewer
??? ???
????
DA/DB
DB
Data
???? ?????A
?????
????
????????
?????
????
????????
GRID?? A
???? ?????C
???? ?????B
Web ???
VO ????
VO? ????
????????
???
VO? Viewer
????

?????
?????
??
GRID?? C
GRID?? B
31
JVO Home Page

• http//jvo.nao.ac.jp/