AstroGrid

1
AstroGrid

• status
• SOFT
• the VO
• the future

AstroGrid presentation to GSC
Andy Lawrence July 2003
2
(No Transcript)
3
status

• six months into Phase B
• team of 26 people (23.4 FTEs)
• spent 1.5M (40 of budget)
• 10,000 lines of code
• working s/w released
• working on infrastructure, standards
• not tools

4
Delivered in Phase-A

• requirements analysis
• technology evaluation
• comprehensive Phase A report
• demonstration s/w (with AVO)
• initial VO standards (with AVO and NVO)
• respected position in UK e-sci programme
• world ranking position in VO work

5
Delivered in Itn-01,02

• basic engine room components
• registry schema, implementation, and query
  service
• data access and job control methods
• MySpace manager, explorer, and registry
• preliminary interface and contents
• simple portal, no tools
• populated with a handful of DCs and datasets
• further international/esci visibility
• use of OGSI, OGSA-DAI, formed GGF-RG
• defined IVOA standards process
• OGSI experiments
• intercontinental visualisation with grid services

6
Planned for Itn03,04

• engine room continued
• consolidate Registry, Data Access, MySpace
• add Auth/Auth, workflow tool (poss Triana)
• Registry harvesting
• interface and contents
• astronomer useable interface
• meaningful contents
• data federation services
• international programme
• standards for Registry, VOQL

7
demonstrations

• Jan 2003 with AVO
• modest technical advance
• new tools, well integrated, excellent PR
• July 2003 AGOC and Sydney
• large technical advance
• no new tools, minimal interface
• next steps
• feedback from AGSAG beta testers crucial
• wrap tools for next demo

8
Target
9
Current
10
(No Transcript)
11
successes

• well motivated hardworking team
• rigorous s/w eng processes
• code cut and released on schedule
• basic engine room components in place
• status in international VO
• status in astronomical community

12
opportunities

• our infrastructure could become standard
• additional tool access via AVO collaboration

13
failures

• portal work failed GUI hacked fixed
• process take-up slow fixed

14
threats

• US diverging on registry standard
• OGSA-DAI vs standard Grid
• too little Data Centres effort
• incorporating datasets
• writing services

15
Progress on AstroGrid goals
(1) develop standards for data, metadata, data
exchange and provenance
GOOD (2) develop a software infrastructure for
data services GOOD (3)
establish a physical grid of resources
GOOD but DC effort worrying (4)
construct and maintain a Service and Resource
Registry GOOD (5)
implement a working VO system of real scientific
use to astronomers TBD
NEXT SIX MONTHS (6) provide a user interface to
that VO system
BEHIND (7) provide or adapt a set of science user
tools to work with the VO
LACK OF FUNDING (8) establish a leading
position for the UK in VO work
GOOD
16
(No Transcript)
17
The VO is a system that will ..

• allows users to interrogate multiple data centres
  in a seamless and transparent way
• provide new powerful analysis and visualisation
  tools within that system
• give data centres a standard framework for
  publishing and delivering services using their
  data.

AstroGrid Phase A Report October 2002
18
multi-l views of a Supernova Remnant
Shocks seen in the X-ray
Dust seen in the IR
Heavy elements seen in the optical
Relativistic electrons seen in the radio
19
What happens to the Earth's magnetosphere during
a coronal mass ejection ?
Event imaged by space-based solar observatory
Effect detected later by satellites and ground
radar
20
needles in a haystack
Hambly et al 2001
- faint moving object is a cool white dwarf - may
be solution to the dark matter problem - but hard
to find one in a million - even harder across
multiple archives
21
The VO is a system that will ..

• allows users to interrogate multiple data centres
  in a seamless and transparent way
• provide new powerful analysis and visualisation
  tools within that system,
• give data centres a standard framework for
  publishing and delivering services using their
  data.

VO-1
VO-2
VO-3
22
The VO is a system that will ..

• allows users to interrogate multiple data centres
  in a seamless and transparent way
• provide new powerful analysis and visualisation
  tools within that system,
• give data centres a standard framework for
  publishing and delivering services using their
  data.

VO-1
looking good
VO-2
almost untouched
VO-3
need effort for DCs to "publish and deliver"
23
the future data services

• VO geometry
• not a warehouse not a hierarchy not P2P
• small set of service centres large popn of
  end-users
• VO content
• not software monolith framework
  standards
• needs content data
  services and user tools
• implies work for Data Centres
• writing services framework uptake resource
  management
• upcoming opportunities internationally
  competitive
• establish a few centres as professionally
  competitive

24
yesterday
web page
CGI request
SQL
DB engine
browser
front end
html
data
25
today
SQL
SOAP/XML request
DB engine
web service
anything
standard formats
application
native data
SOAP/XML data
26
tomorrow
web service
web service
publish WSDL
web service
job
Registry Workflow GLUE AstroPass
MySpace
web service
standard semantics
anything
application
results
web service
web service
27
day after tomorrow
grid service
grid service
pooled resource
ontology
grid service
job
Registry Workflow GLUE AstroPass
MySpace
agents
standard semantics
grid service
anything
application
results
grid service
grid service
28
work needed
grid service
grid service
TECHNOLOGY RESEARCH
pooled resource
TOOLS
ontology
grid service
job
Registry Workflow GLUE AstroPass
MySpace
DATA PIPELINES
agents
standard semantics
grid service
anything
application
results
grid service
STANDARDS
DATA SERVICES (access and analysis)
grid service
INFRASTRUCTURE
GRID UPTAKE
29
(No Transcript)
30
expected status 2004

• tools ok but minimal
• standards good but evolving
• glue good but could be better
• data access services ok but under-resourced
• data analysis services almost untouched
• grid uptake ready to deploy but resource worry
• technology research treading water

31
AstroGrid-2

• priority areas
• core evolve and improve, build tools
• establish content data services, resource grid
• analysis services visualisation, datamining
  research and deploy
• next technology wave ontology, agents
• expertise needed
• existing AstroGrid developer team
• alliance of Data Centres
• add Computer Science expertise

32
CS expertise

• Leeds visualisation experts
• Portsmouth kd-tree algorithms
• Exeter Intelligent Agents for astronomy
• commercial experience Linde, Noddle
• local links Bristol, Edinburgh, Cambridge
• external links OGSA-DAI, GGF, MS, IBM

33
Infrastructure

• framework largely in place but..
• needs to be robust and reliable
• needs continuing work on standards
• IVOA and Grid
• needs suite of tools
• key idea is personalisation
• tool requests from competitive calls

delivers VO-1
34
RD

• really mostly D
• assess, customise, integrate
• keep on the next wave
• ontology, agents
• keep promise that everybody can be a power user
• large volume and multi-D visualisation and
  datamining services

delivers VO-2 and re-newed VO-1
35
Data Centres

• Data Centres are the heart of the VOthere is no
  UK VO without well resourced DCs
• top priority is support for VO/uptake
• second priority is core support for DCs
• not mission specific resources
• not pipeline or archive development
• basic physical resource management and data
  curation

delivers VO-1
delivers VO-3
36
Costed Model Options

• M1 full programme 9.25M
• M2 full research 6.68M
• lose DCA core support
• world beating VO infrastructure but UK content
  compromised
• M3 partial research 5.28M
• lose some lines and trim others minimise new
  algorithms
• good working VO but loses opportunity for UK lead
  in new technology
• approximately flat continuation of AG1-PhaseB
• M4 minimum viable 3.75M
• no research unlikely to lead any third wave of
  VO development
• working VO but relies on external components

37
(No Transcript)
38
FIN