Distributed Databases and Applications

1
Distributed Databases and Applications

• John Wieczorek
• Museum of Vertebrate Zoology, UC Berkeley

2
Distributed Databases

• Multiple sources of data
• under local control,
• with concepts in common
• and a desire to deliver data as part of a
  community.

3
Distributed Databases

• The Species Analyst (TSA)
• The Integrated Taxonomic Information System
  (ITIS)
• FishNet
• The Mammal Networked Information System (MaNIS)
• HerpNET
• The Ornithological Information System (ORNIS)

4
Distributed Databases

• European Natural History Science Information
  Network (ENHSIN)
• Biological Collection Access for Europe (BioCASE)
• Australia Virtual Herbarium (AVH)
• Red Mundial de Información Sobre Biodiversidad,
  Comisión Nacional para el Conocimiento y Uso de
  la Biodiversidad (REMIB, CONABIO)

5
Distributed Databases

• Mountain and Plains Spatio-Temporal
  Database-Informatics (MaPSTeDI)
• Ocean Biogeographic Information System (OBIS)
• Pacific Basin Information Node, National
  Biological Information Infrastructure (PBIN,
  NBII)
• Species Link, Centro de Referência em Informação
  Ambiental (Species Link, CRIA)
• A Virtual Herbarium of the Chicago Region
  (vPlants)
• Spatial Analysis of Local Vegetation Inventories
  Across Scales (SALVIAS)

6
Distributed Databases

• Berkeley Natural History Museums (BNHM)
• Association of Biological Collections, UC Davis

7
Distributed Databases

• LifeMapper
• Global Biodiversity Information Facility (GBIF)

8
Distributed vs. centralized

• Multiple sources of data
• under local control,
• with concepts in common
• and a desire to deliver data as part of a
  community

9
Distributed vs. centralized

• In other words, distribute the headache rather
  than have one central migraine.

10
DiGIRDistributed Generic Information Retrieval

• John Wieczorek, Stan Blum, Dave Vieglais, P.J.
  Schwartz

11
Project Rationale

• To avoid multiple incongruous development efforts
• To pool resources and create a community of
  experts
• To solve the problem of scalability

12
Project Goals

• To define a protocol for retrieving structured
  data from multiple, heterogeneous databases
  across the Internet
• To build a reference implementation of both
  provider and portal software using said protocol

13
Design Goals

• To use open protocols and standards, such as HTTP
  and XML
• To decouple the protocol, software and semantics
• To make new data provider installations as easy
  as possible
• To have open source development and GNU General
  Public Licensing

14
DiGIR Architecture

• User Interface
• Protocol
• Portal Engine
• Provider

15
DiGIR Architecture

• Provider

16
DiGIR Architecture

• Provider
• Registry

17
DiGIR Architecture

• Portal Engine

18
DiGIR Architecture

• Portal Engine
• Registry

19
DiGIR Architecture

• User Interface

20
DiGIR Architecture

• User Interface
• Protocol
• Portal Engine

21
DiGIR Architecture

• User Interface
• Protocol
• Portal Engine
• Protocol
• Provider

22
DiGIR Architecture

• User Interface
• Protocol
• Portal Engine
• Protocol
• Provider

23
DiGIR Architecture

• User Interface
• Protocol
• Portal Engine

24
DiGIR Component Summary
25
DiGIR Protocol

• Defines request and response message formats for
  communication between provider, portal engine,
  and user interfaces
• Metadata requests
• Search requests
• Inventory requests
• Remains unfettered by the structure of the data
  it transfers

26
Portal Engine

• The entry point for a user
• Can query a registry for potential providers
• Can determine, based on provider metadata,
  whether a provider should be queried
• Can send requests to multiple providers
• Communicates via protocol compliant messaging only

27
Portal Engine, continued

• Assembles responses from providers
• Returns packaged results to the user
• Logs activity

28
Provider

• Receives requests
• Retrieves data from database
• Sends results to requestor
• Supplies metadata to describe data classification
  and availability
• Logs requests

29
Registry

• Supports provider advertising
• May be global and open
• May be private
• Need not be used at all
• Example Universal Description, Discovery and
  Integration (UDDI)

30
User Interfaces

• Must be able to assemble and send a request
  document to a portal
• Must be able to receive and interpret a response
  document from the portal
• This is where the real fun is!

31
Example Network Configurations
32
BNHM Network Configuration
BNHM DiGIR Portal
BNHM Presentation Layer
33
MaNIS Network Configuration
MaNIS DiGIR Portal
MaNIS DiGIR Portal
MaNIS DiGIR Portal
MaNIS Presentation Layer
MaNIS Presentation Layer
MaNIS Presentation Layer
34
MaNIS Network Configuration
MaNIS DiGIR Portal
MaNIS DiGIR Portal
MaNIS DiGIR Portal
MVZ-MaNIS Presentation Layer
LACM-MaNIS Presentation Layer
UWBM-MaNIS Presentation Layer
35
MaNIS Network Configuration
MaNIS DiGIR Portal
MaNIS DiGIR Portal
MaNIS DiGIR Portal
MVZ-MaNIS Presentation Layer
LACM-MaNIS Presentation Layer
UWBM-MaNIS Presentation Layer
36
MaNIS Network Configuration
MaNIS DiGIR Portal
MaNIS DiGIR Portal
MaNIS DiGIR Portal
MVZ-MaNIS Presentation Layer
LACM-MaNIS Presentation Layer
UWBM-MaNIS Presentation Layer
37
MaNIS Network Configuration
MaNIS DiGIR Portal
MaNIS DiGIR Portal
MaNIS DiGIR Portal
MVZ-MaNIS Presentation Layer
LACM-MaNIS Presentation Layer
UWBM-MaNIS Presentation Layer
38
MaNIS Network Configuration
MaNIS DiGIR Portal
MaNIS DiGIR Portal
MaNIS DiGIR Portal
MVZ-MaNIS Presentation Layer
LACM-MaNIS Presentation Layer
UWBM-MaNIS Presentation Layer
39
Other Network Configurations
40
DiGing a little deeper
41
Provider Installation

• Web server (Apache, IIS, etc.)
• PHP Hypertext Preprocessor (PHP)
• Provider software (DiGIR)
• Configuration tool
• Testing scripts
• Provider scripts
• Provider manual (DiGIR)

42
Provider Configuration Tool

• Provider metadata
• Resources
• Database connection
• Establishing table relationships
• Concept to column (i.e., field, attribute) mapping

43
Portal Configuration

• Web server (Apache, IIS, etc.)
• Sun Java 2 (JDK 1.4)
• Tomcat (Apache)
• Portal software (DiGIR)
• Portal installation documentation (DiGIR)

44
Portal Installation

• Engine configuration file (finding providers)
• Presentation configuration file (defining the
  Information Domain)
• Presentation customization
• Engine start and stop scripts
• Presentation start and stop scripts

45
Portal Demonstrations
46
DiGIR Project Information

• The DiGIR project is a collaborative effort
• DiGIR is currently established as an open source
  development project on SourceForge
  (https//sourceforge.net/projects/digir).
• Further documentation is available on the DiGIR
  web site (http//digir.net).