ADaM%20System%20Architecture

1
ADaM System Architecture

• Rahul Ramachandran, Sara Graves and
• Ken Keiser
• Mathematical Challenges in Scientific Data Mining
• IPAM January 14-18, 2002
• Information Technology and Systems Center
• University of Alabama in Huntsville
• rramachandran_at_itsc.uah.edu

2
Talk Overview

• Mining System Requirements
• ADaM System Architecture
• ADaM Plan Builder
• Research directions

3
Mining System Requirements When,Where and Who

• WHERE
• User Workstation
• Data Archive Center
• Data Mining Center

• WHEN
• Real Time
• On-Ingest
• On-Demand
• Repeatedly

• WHO
• Casual Users
• Domain Experts
• Mining Experts

Data Mining
4
Algorithm Development and Mining (ADaM) System

• ADaM system developed under NASA research grant
• The system provides knowledge discovery, feature
  detection and content-based searching for data
  values, as well as for metadata.
• It contains over 120 different operations to be
  performed on the input data stream.
• Operations vary from specialized atmospheric
  science data-set specific algorithms to different
  digital image processing techniques, processing
  modules for automatic pattern recognition,
  machine perception, neural networks and genetic
  algorithms.

5
ADaM Features

• Handles science data set variability
• Multiple resolution/multiple scales
• Variability of formats
• Granularity of data
• Includes spatial/temporal dimensions
• Allows addition of new algorithms
• Allow scientists to select and sequence different
  operations

6
ADaM Engine Architecture
Preprocessed Data
Patterns/ Models
Results
Data
Translated Data
Processing

Preprocessing
Analysis
Selection and Sampling Subsetting
Subsampling Select by Value Coincidence
Search Grid Manipulation Grid Creation
Bin Aggregate Bin Select Grid Aggregate
Grid Select Find Holes Image Processing
Cropping Inversion Thresholding Others...
Clustering K Means Isodata
Maximum Pattern Recognition Bayes Classifier
Min. Dist. Classifier Image Analysis
Boundary Detection Cooccurrence Matrix
Dilation and Erosion Histogram Operations
Polygon Circumscript Spatial Filtering
Texture Operations Genetic Algorithms Neural
Networks Others...
7
ADaM Mining Environment
Data Mining Server
Mining Results
Event/ Relationship Search System
8
ADaM Architecture
9
ADaM Miner Engine

• Manages the processing of data through a series
  of specified operations
• Loads input, processing and output modules
  dynamically as needed at execution time
• Allows for the addition of newly developed
  modules without the need to rebuild the engine
• Interprets a mining plan script that provides the
  details about specified operations and the order
  that they should be executed

10
ADaM Miner Database

• Used to store information that includes the
  names, locations and related metadata for input
  data sets available on the server
• Includes information about users, jobs, mining
  results, and other related information
• Simple relational database

11
ADaM Daemon and Scheduler

• Scheduler
• Examines the list of jobs to be executed on the
  server and determines which job or jobs to
  execute at any given time
• Queues the requests and executes them
  sequentially.
• Daemon
• Handles all network communications with the
  mining system
• Is configured to listen on a specific port for
  any socket communications

12
ADaM Input/Operation Filters

• Input/Output Filters are data readers and writers
• Operations are the algorithms
• Each of the operations and (input/output)
  filters is implemented as a shared library
• New modules may be added to the system without
  recompiling or relinking.
• All operations/filters either produce or operate
  on a data collection, which provides a common
  format for representing scientific data.

13
General Mining Steps

• Select data files to be mined
• Check-In the data files into the Miner Database
• Write a Mining Plan consisting of sequence of
  input filter and operations
• Execute the Mining Plan using the engine
• Check and save results
• Iterate

14
What is Check-In?

• Process of encoding information such as the
  names, locations and related metadata for input
  data sets available on the server
• Create complex data hierarchy in the database

15
ADaM Plan Builder Check-In

• Two Modes of Operation
• General which only requires
• minimal information
• Advanced requires more
• detailed information and
• Allows user to set up
• structured database

Path to the data file
Data file name
Input Filter associated with the Data file
Load an XML file containing existing Check-In
specifications
16
ADaM Plan Builder Layout
Operation Menu contains the list of operations
one can select
Input Menu contains the list of Input Filters one
can select

• Plan Menu allows one to
• Select a new plan
• Load existing plan
• Check-In data

17
ADaM Plan Builder Layout
Panel where Mining Plan can be viewed either as
text or a tree
18
ADaM Plan Builder Layout
Description about the Operation/Input Filter can
be viewed in this panel
19
ADaM Plan Builder Layout
All the parameters needed for the Operation are
described here
20
ADaM Plan Builder Layout
Sample values for Operations parameters are
shown in this panel
21
ADaM Plan Builder Layout
Allows user to select the operation and add it to
the Mining Plan
Go Mine the data using the Mining Plan
22
Research Directions

• Generic Data Reader for ADaM
• ESML Earth Science Markup Language
• Programmers Guide for ADaM
• Distributed Mining
• Grid Mining
• Successful implementation and testing of the ADaM
  system on the NASA Information Power Grid
• Mining Onboard the Space Craft
• The EnVironmEnt for On-Board Processing (EVE)
  system

23
ADaM Information

• Web site
• datamining.itsc.uah.edu
• ADaM Lite beta version download
• Contact rramachandran_at_itsc.uah.edu