The World Wide Telescope

1
The World Wide Telescope a Digital Library
Prototype

• Jim Gray, Microsoft Research
• Alex Szalay, Johns Hopkins University

Talk at OCLC _at_ Dublin, OH, 17 May
2004 http//research.microsoft.com/gray/talks/OCL
C_WWT.ppt
2
Jims Model of Library Science ?

• Alexandria
• Gutenberg
• (Melvil) Dewey Decimal
• MARC (Henriette Avram)
• Dublin Core

Yes, I know there have been other things.
3
Dublin Core

• Elements
• Title
• Creator
• Subject
• Description
• Publisher
• Contributor
• Date
• Type
• Format
• Identifier
• Source
• Language
• Coverage
• Rights

• Elements
• Audience
• Alternative
• TableOfContents
• Abstract
• Created
• Valid
• Available
• Issued
• Modified
• Extent
• Medium
• IsVersionOf
• HasVersion
• IsReplacedBy
• Replaces
• IsRequiredBy
• Requires
• IsPartOf

• Encoding
• LCSH (Lb. Congress Subject Head)
• MESH (Medical Subject Head)
• DDC (Dewey Decimal Classification)
• LCC (Lb. Congress Classification)
• UDC (Universal Decimal Classification)
• DCMItype (Dublin Core Meta Type)
• IMT (Internet Media Type)
• ISO639-2 (ISO language names)
• RFC1766 (Internet Language tags)
• URI (Uniform Resource Locator)
• Point (DCMI spatial point)
• ISO3166 (ISO country codes)
• Box (DCMI rectangular area)
• TGN (Getty Thesaurus of Geo Names)
• Period (DCMI time interval)
• W3CDTF (W3C date/time)
• RFC3066 (Language dialects)

Thanks!
4
Whats Happening?

• We are drowning in information
• Single fixed hierarchy is hopeless
• Cant organize/find things in a simple tree
• HOPE schematized storage
• Objects have Dublin-like facets
• Most facets acquired automatically (email, photo,
  doc,)
• Users add annotations and relationships
  Librarians call this accession
• Automate accession as much as possible
• Folders/directories are standing queries
• Organization is search based demo sis.
• Interesting (public) research projects
• Stuff Ive Seen http//research.microsoft.com/ada
  pt/sis/
• MyLifebits http//research.microsoft.com/barc/med
  iapresence/MyLifeBits.aspx
• Longhorn product embraces extends these ideas.

5
The World Wide Telescope a Digital Library
Prototype
But, what about the talk I promised you?

• Jim Gray, Microsoft Research
• Alex Szalay, Johns Hopkins University

Talk at OCLC _at_ Dublin, OH, 17 May
2004 http//research.microsoft.com/gray/talks/OCL
C_WWT.ppt
6
The Talk

• Libraries morphing to integrated text data
  (you know that)
• Dublin Core is bedrock, but many issues remain.
  (you know that)  
• WWT All Astronomy data and literature online and
  integrated
• Problems Librarians have grappled with for
  centuries curation, preservation, indexing,
  access, summarization.  
• Overview of the World-Wide Telescope as a digital
  library
• Focus on metadata, schema, curation, and
  preservation..
• Candidly, we have more problems than solutions,
  but the data is arriving and we are doing the
  best we can.

7
New Science Paradigms

• Thousand years ago science was empirical
• describing natural phenomena
• Last few hundred years theoretical branch
• using models, generalizations
• Last few decades a computational branch
• simulating complex phenomena
• Today data exploration (eScience)
• synthesizing theory, experiment and
  computation with advanced data management and
  statistics

8
The Big Picture
Experiments Instruments
facts
questions
?
facts
Other Archives
answers
facts
Literature
facts
Simulations
The Big Problems

• Data ingest
• Managing a petabyte
• Common schema
• How to organize it?
• How to reorganize it
• How to coexist with others

• Data Query and Visualization tools
• Support/training
• Performance
• Execute queries in a minute
• Batch (big) query scheduling

9
The Virtual Observatory

• Premise most data is (or could be online)
• The Internet is the worlds best telescope
• It has data on every part of the sky
• In every measured spectral band optical, x-ray,
  radio..
• As deep as the best instruments (2 years ago).
• It is up when you are up
• The seeing is always great
• Its a smart telescope links objects and
  data to literature
• Software is the capital expense
• Share, standardize, reuse..

10
Why Is Astronomy Special?

• Almost all literature online and public ADS
  http//adswww.harvard.edu/ CDS
  http//cdsweb.u-strasbg.fr/
• Data has no commercial value
• No privacy concerns, freely share results with
  others
• Great for experimenting with algorithms
• It is real and well documented
• High-dimensional (with confidence intervals)
• Spatial, temporal
• Diverse and distributed
• Many different instruments from many different
  places and many different times
• The community wants to share the data
• There is a lot of it (soon petabytes)

11
Like all sciences, Astronomy Faces an
Information Avalanche

• Astronomers have a few hundred TB now
• 1 pixel (byte) / sq arc second 4TB
• Multi-spectral, temporal, ? 1PB
• They mine it looking for new (kinds of) objects
  or more of interesting ones (quasars),
  density variations in 400-D space correlations
  in 400-D space
• Data doubles every year
• Data is public after 1 year
• So, 50 of the data is public
• Same access for everyone

12
Publishing Data

• Exponential growth
• Projects last at least 3-5 years
• Data sent upwards only at the end of the project
• Data will never be centralized
• More responsibility on projects
• Becoming Publishers and Curators
• Data will reside with projects
• Analyses must be close to the data

13
How to Publish Data Web Services

• Web SERVER
• Given a url parameters
• Returns a web page (often dynamic)
• Web SERVICE
• Given a XML document (soap msg)
• Returns an XML document (with schema)
• Tools make this look like an RPC.
• F(x,y,z) returns (u, v, w)
• Distributed objects for the web.
• naming, discovery, security,..
• Internet-scale distributed computing

Your program
Web Server
http
Web page
Your program
Web Service
soap
Data In your address space
objectin xml
14
The Core Problem No Economic Model

• The archive user has not yet been born. How can
  he pay you to curate the data?
• Q The Scientist gathered data for his own
  purpose.Why should he pay (invest time) for your
  needs?A thats the scientific method
• Curating data (documenting the design, the
  acquisition, and the processing)is very hard and
  there is no reward for doing it.Results are
  rewarded, not the process of getting them.
• Storage/archive NOT the problem (its almost
  free)
• Curating/Publishing is expensive.
• Better standards tools lower costs

15
Data Inflation Data Pyramid

• Level 1AGrows 5TB pixels/year growing to
  25TB 2 TB/y compressed growing to 13TB 4
  TB today (level 1A in NASA terms)

• Level 2Derived data products 10x smaller But
  there are many catalogs.
• Publish new edition each year
• Fixes bugs in data.
• Must preserve old editions
• Creates data pyramid
• Store each edition
• 1, 2, 3, 4 N N2 bytes
• Net Data Inflation L2 L1

16
What SDSS is Doing Capture the Bits

• Best-effort documenting data and process.
• Publishing data often by UPS( 5TB today and so
  5k for a copy)
• Replicating data on 3 continents.
• EVERYTHING online (tape data is dead data)
• Archiving all email, discussions, .
• Keeping all web-logs.
• Now we need to figure out how to organize/search
  all this metadata.

17
Making Discoveries

• Where are discoveries made?
• At the edges and boundaries
• Going deeper, collecting more data, using more
  colors.
• Metcalfes law quadratic benefit
• Utility of computer networks grows as the number
  of possible connections O(N2)
• Data Federation quadratic benefit
• Federation of N archives has utility O(N2)
• Possibilities for new discoveries grow as O(N2)
• Current sky surveys have proven this
• Very early discoveries from SDSS, 2MASS, DPOSS

18
Global Federations

• Massive datasets live near their owners
• Near the instruments software pipeline
• Near the applications
• Near data knowledge and curation
• Each Archive publishes a (web) service
• Schema documents the data
• Methods on objects (queries)
• Scientists get personalized extracts
• Uniform access to multiple Archives
• A common global schema

Federation
19
Schema (aka metadata)

• Everyone starts with the same schema
  ltstuff/gtThen the start arguing about semantics.
• Virtual Observatory http//www.ivoa.net/
• Metadata based on Dublin Corehttp//www.ivoa.net
  /Documents/latest/RM.html
• Universal Content Descriptors (UCD)
  http//vizier.u-strasbg.fr/doc/UCD.htxCaptures
  quantitative concepts and their unitsReduced
  from 100,000 tables in literature to 1,000
  terms
• VOtable a schema for answers to
  questionshttp//www.us-vo.org/VOTable/
• Common QueriesCone Search and Simple Image
  Access Protocol, SQL
• Registry http//www.ivoa.net/Documents/latest/RME
  xp.htmlstill a work in progress.

20
Data Access is Hitting a Wall
Current practice of data download (FTP/GREP)will
not scale to petabyte datasets

• You can FTP 1 MB in 1 sec
• You can FTP 1 GB / min ( 1 /GB)
• You can FTP 1 TB in 2 days and 1K
• You can FTP 1 PB in 3 years and 1M

• You can GREP 1 MB in a second
• You can GREP 1 GB in a minute
• You can GREP 1 TB in 2 days
• You can GREP 1 PB in 3 years
• Oh!, and 1PB 4,000 disks
• At some point you need indices to limit
  search parallel data search and analysis
• This is where databases can help

21
Smart Data

• Better Data Schemas
• There is too much data to move aroundDo data
  manipulations at database
• Build custom procedures and functions into DB
• Unify data Access Analysis
• Examples
• Temporal and spatial indexing
• Pixel processing
• Automatic parallelism
• Auto (re)organize
• Scalable to Petabyte datasets

Move Mohamed to the mountain, not the mountain to
Mohamed.
22
Next-Generation Data Analysis

• Looking for
• Needles in haystacks the Higgs particle
• Haystacks dark matter, dark energy, turbulence,
  ecosystem dynamics
• Needles are easier than haystacks
• Global statistics have poor scaling
• Correlation functions are N2, likelihood
  techniques N3
• As data and computers grow at Moores Law, we
  can only keep up with N logN
• A way out?
• Relax optimal notion (data is fuzzy, answers are
  approximate)
• Dont assume infinite computational resources or
  memory
• Requires combination of statistics computer
  science

23
The Sloan Digital Sky Survey

• Goal
• Create the most detailed map of the Northern Sky
  to-date
• 2.5m telescope
• 3 degree field of view
• Two surveys in one
• 5-color images of ¼ of the sky
• Spectroscopic survey of a million galaxies and
  quasars
• Very high data volume
• 40 Terabytes of raw data
• 10 Terabytes processed
• All data public

The University of Chicago Princeton
University The Johns Hopkins University The
University of Washington New Mexico State
University University of Pittsburgh Fermi
National Accelerator Laboratory US Naval
Observatory The Japanese Participation Group
The Institute for Advanced Study Max Planck
Inst, Heidelberg Sloan Foundation, NSF, DOE,
NASA
24
SkyServer

• A multi-terabyte database
• An educational website
• More than 50 hours of educational exercises
• Background on astronomy
• Tutorials and documentation
• Searchable web pages
• Easy astronomer access to SDSS data.
• Prototype eScience lab
• Interactive visual tools for data exploration

http//skyserver.sdss.org/
25
Demo SkyServer

• atlas
• education project
• Mouse in pixel space
• Explore an object (record space)
• Explore literature
• Explore a set
• Pose a new question

26
SkyQuery (http//skyquery.net/)

• Distributed Query tool using a set of web
  services
• Many astronomy archives from Pasadena, Chicago,
  Baltimore, Cambridge (England)
• Has grown from 4 to 15 archives,now becoming
  international standard
• Allows queries like

SELECT o.objId, o.r, o.type, t.objId FROM
SDSSPhotoPrimary o, TWOMASSPhotoPrimary t
WHERE XMATCH(o,t)lt3.5 AND AREA(181.3,-0.76,6.5)
AND o.type3 and (o.I - t.m_j)gt2
27
Demo SkyQuery Structure

• Portal is
• Plans Query (2 phase)
• Integrates answers
• Is itself a web service

• Each SkyNode publishes
• Schema Web Service
• Database Web Service

28
MyDB eScience Workbench

• Prototype of bringing analysis to the data
• Everybody gets a workspace (database)
• Executes analysis at the data
• Store intermediate results there
• Long queries run in batch
• Results shared within groups
• Only fetch the final results
• Extremely successful matches work patterns

29
National Center Biotechnology Information (NCBI)
A Better Example

• Pubmed
• Abstracts and books and..
• Genbank
• All Gene sequences deposited
• BLAST and other searches
• Website to explore data and literature
• Entrez
• unifies many databases with literature (books,
  journals,..)
• Organizes the data

30
The Big Picture
Experiments Instruments
facts
questions
?
facts
Other Archives
answers
facts
Literature
facts
Simulations
The Big Problems

• Data ingest
• Managing a petabyte
• Common schema
• How to organize it?
• How to reorganize it
• How to coexist with others

• Query and Vis tools
• Support/training
• Performance
• Execute queries in a minute
• Batch query scheduling

31
The Talk

• Libraries morphing to integrated text data
  (you know that)
• Dublin Core is bedrock, but many issues remain.
  (you know that)  
• WWT All Astronomy data and literature online and
  integrated
• Problems Librarians have grappled with for
  centuries curation, preservation, indexing,
  access, summarization.  
• Overview of the World-Wide Telescope as a digital
  library
• Focus on metadata, schema, curation, and
  preservation..
• Candidly, we have more problems than solutions,
  but the data is arriving and we are doing the
  best we can.

32
Education

• Educational Projects, aimed at advanced high
  school students, but covering middle school
• Teach how to analyze data, discover patterns,not
  just astronomy
• 3.7 million project hits, 1.25 million page
  views of educational content
• More than 4000 textbooks
• On the whole web site 44 million web hits
• Largely a volunteer effort by many individuals
• Matches the 2020 curriculum