Distributed Databases and Applications

1
Distributed Databases and Applications

• John Wieczorek
• Museum of Vertebrate Zoology, UC Berkeley

2
Warning

• What you are about to witness is not meant to be
  experienced in a single dose.
• Side effects may include nausea, eye twitching,
  and general discomfort.
• Do not panic, call for help.

3
DiGIRDistributed Generic Information Retrieval

• John Wieczorek, Stan Blum, Dave Vieglais, P.J.
  Schwartz

4
Information Retrieval

• Distributed - a protocol for retrieving
  structured data from multiple, heterogeneous
  databases across the Internet.
• Generic - a protocol independent of the data
  retrieved and of the software to retrieve it.

5
Project Rationale

• Avoid multiple incongruous development efforts
• Pool resources and create a support community of
  experts
• Solve scalability problems

6
Design Goals

• Use open protocols and standards, such as HTTP
  and XML
• Decouple the protocol, software and semantics
• Make new data provider installations as easy as
  possible
• Develop open source software with GNU General
  Public Licensing (Its free).

7
DiGIR Component Summary
8
DiGIR Architecture

• Provider

9
Provider

• Receives requests
• Retrieves data from database
• Sends results to requestor
• Supplies metadata to describe content, contacts,
  and capabilities
• Logs requests

10
DiGIR Architecture

• Portal Engine

11
Portal Engine

• The entry point for an application
• Can query a registry to discover potential
  providers
• Can determine, based on provider metadata,
  whether a provider should be queried
• Can send requests to multiple providers

12
Portal Engine, continued

• Assembles responses from providers
• Returns packaged results to the requesting
  application
• Communicates via protocol compliant messaging
  only
• Logs activity

13
Registry

• Provides a yellow pages to advertise the
  existence and capabilities of a provider
• Provides a means to discover potential providers
  of interest
• May be public or private
• Need not be a part of the architecture

14
DiGIR Architecture

• Provider
• Registry (register)

15
DiGIR Architecture

• Portal Engine
• Registry (discover)

16
DiGIR Protocol

• Defines request and response message formats for
  communication between provider, portal engine,
  and applications
• Metadata requests
• Search requests
• Inventory requests
• Remains unfettered by the structure of the data
  it transfers

17
DiGIR Architecture

• Application

18
DiGIR Architecture

• Application
• Protocol (request)
• Portal Engine

19
DiGIR Architecture

• Application
• Protocol (request)
• Portal Engine
• Protocol (request)
• Provider

20
DiGIR Architecture

• Application
• Protocol (request)
• Portal Engine
• Protocol (response)
• Provider

21
DiGIR Architecture

• Application
• Protocol (response)
• Portal Engine

22
Applications

• Must be able to assemble and send a request
  document to a portal
• Must be able to receive and interpret a response
  document from the portal
• Must do something incredibly useful and
  interesting with the data
• This is where the real fun is!

23
MaNIS The Mammal Networked Information System
Its more than just a pangolin
24
MaNIS Network Configuration
MaNIS DiGIR Portal
MaNIS DiGIR Portal
MaNIS DiGIR Portal
MVZ-MaNIS Presentation Layer
UMNH-MaNIS Presentation Layer
UWBM-MaNIS Presentation Layer
25
MaNIS Network Configuration
MaNIS DiGIR Portal
MaNIS DiGIR Portal
MaNIS DiGIR Portal
MVZ-MaNIS Presentation Layer
UMNH-MaNIS Presentation Layer
UWBM-MaNIS Presentation Layer
26
MaNIS Network Configuration
MaNIS DiGIR Portal
MaNIS DiGIR Portal
MaNIS DiGIR Portal
MVZ-MaNIS Presentation Layer
UMNH-MaNIS Presentation Layer
UWBM-MaNIS Presentation Layer
27
MaNIS Network Configuration
MaNIS DiGIR Portal
MaNIS DiGIR Portal
MaNIS DiGIR Portal
MVZ-MaNIS Presentation Layer
UMNH-MaNIS Presentation Layer
UWBM-MaNIS Presentation Layer
28
MaNIS Network Configuration
MaNIS DiGIR Portal
MaNIS DiGIR Portal
MaNIS DiGIR Portal
MVZ-MaNIS Presentation Layer
UMNH-MaNIS Presentation Layer
UWBM-MaNIS Presentation Layer
29
MaNIS Network Configuration
MaNIS DiGIR Portal
MaNIS DiGIR Portal
MaNIS DiGIR Portal
MVZ-MaNIS Presentation Layer
UMNH-MaNIS Presentation Layer
UWBM-MaNIS Presentation Layer
30
MaNIS Network Configuration
MaNIS DiGIR Portal
MaNIS DiGIR Portal
MaNIS DiGIR Portal
MVZ-MaNIS Presentation Layer
UMNH-MaNIS Presentation Layer
UWBM-MaNIS Presentation Layer
31
MaNIS Network Configuration
MaNIS DiGIR Portal
MaNIS DiGIR Portal
MaNIS DiGIR Portal
MVZ-MaNIS Presentation Layer
UMNH-MaNIS Presentation Layer
UWBM-MaNIS Presentation Layer
32
MaNIS Network Configuration
MaNIS DiGIR Portal
MVZ-MaNIS Presentation Layer
33
MaNIS Network Configuration
MaNIS DiGIR Portal
MVZ-MaNIS Presentation Layer
34
MaNIS Network Configuration
MaNIS DiGIR Portal
MVZ-MaNIS Presentation Layer
35
MaNIS Network Configuration
MaNIS DiGIR Portal
MVZ-MaNIS Presentation Layer
36
MaNIS Network Configuration
MaNIS DiGIR Portal
MVZ-MaNIS Presentation Layer
37
MaNIS Network Configuration
MaNIS DiGIR Portal
MVZ-MaNIS Presentation Layer
38
MaNIS Network Configuration
MaNIS DiGIR Portal
MVZ-MaNIS Presentation Layer
39
MaNIS Network Configuration
MaNIS DiGIR Portal
MVZ-MaNIS Presentation Layer
40
MaNIS Network Configuration
MaNIS DiGIR Portal
MVZ-MaNIS Presentation Layer
41
MaNIS Network Configuration
MaNIS DiGIR Portal
MVZ-MaNIS Presentation Layer
42
MaNIS Network Configuration
MaNIS DiGIR Portal
MVZ-MaNIS Presentation Layer
43
MaNIS Network Configuration






MaNIS DiGIR Portal
MaNIS DiGIR Portal
MaNIS DiGIR Portal


MVZ-MaNIS Presentation Layer
UMNH-MaNIS Presentation Layer
UWBM-MaNIS Presentation Layer
44
MaNIS Network Configuration
CalNet DiGIR Portal
45
MaNIS Network Configuration
CalNet DiGIR Portal
BioGeomancer Web Service
46
MaNIS Network Configuration
NBII DiGIR Portal
47
MaNIS Network Configuration
NBII DiGIR Portal
GBIF Presentation Layers
48
Intra-Network Configuration (BNMH)
BNHM DiGIR Portal
BNHM Presentation Layer
49
Other Network Configurations
50
Other Network Configurations
51
Other Network Configurations
52
Other Network Configurations
53
Other Network Configurations
54
Other Network Configurations
55
Distributed vs. centralized

• Multiple sources of data
• under local control,
• with concepts in common
• and a desire to deliver as part of a community

56
Distributed vs. centralized

• In other words, distribute the headache avoid
  migraines.

57
Project Information

• DiGIR is a collaborative open source development
  project on SourceForge (https//sourceforge.net/pr
  ojects/digir).
• Software and documentation are available on the
  DiGIR web site (http//digir.net).
• MaNIS is an international network collaboration
  among mammal specimen collections
  (http//elib.cs.berkeley.edu/manis).

58
Portal Demonstrations