IMASD Design

1
IMAS-DImage Management System Demonstrator Proje
ct Overview (extracts from presentations given at
the official project meetings) M.
Verola massimo.verola_at_roma.quadrics.com
2

• Anatomy of the Project

3
Application Scenario
Remote Sensing (RS) data and related products
have proven they can be exploited not only in
scientific and academic communities, but also in
governmental institutions, in national defense
bodies and in commercial enterprises, thanks to
the continuous increasing of spatial and spectral
data resolution. In particular SAR, panchromatic
and hyperspectral sensors are becoming more and
more a source of information of paramount
importance for a variety of critical applications
such as environmental protection, urban and
regional planning and monitoring, agricultural
census, military tactical and strategic
reconnaissance and surveillance.

• Anatomy of the Project

4
Need for HPC
As higher resolution (? 1 meter) data become
available and the provision rate of data
increases, due to the availability of satellite
constellation with high revisit frequency, an
enormous amount of data will have to be processed
and stored. The most effective way to face to
this huge amount of information is to manage them
by using high performance computing facilities
and appropriate parallel algorithms.

• Anatomy of the Project

5
Project Objectives
Main Goal validate and accelerate the
application of High Performance Computing (HPC)
to command, control, communication and
intelligence systems, in particular in the field
of Image Management and Processing
systems. Technical Objective realize an Imagery
Management and Processing System able to
demonstrate its relevance by means of - the
provision of advanced image processing
functionality, and, in particular, of parallel
algorithms - the management (storage, retrieval,
exploitation) of imagery data, linearly scalable
to a potential unlimited size without performance
degradation - the integration of a multi-tier
software system based on both commercial packages
and specialized newly developed modules, where a
powerful parallel processing engine, a
state-of-the-art image visualization environment,
an innovative DBMS and a Web-based product
dissemination gateway cooperate together for
building up an end-to-end RS data management
system

• Anatomy of the Project

6
Project Exploitation
The main benefit of combining the power of
cluster computing with the advanced functionality
of state-of-the-art commercial software packages
is the opportunity for the end user to greatly
expand the set of problems that can be solved,
both in terms of computational domain size than
in terms of the quality of the implemented
algorithms and techniques.   The prospected
product resulting from the evolution of IMAS-D
will match the ever increasing computing power
and memory resource demands that new-generation
RS sensors, especially in the hyperspectral
field, will require for an effective exploitation
of the produced raw data.

• Anatomy of the Project

7
Technical Risks

• "The challenge of the IMAS-D project is to create
  a framework where stable commercial components
  live together with innovative components, thus
  several technical risks are implicit in the
  design of the system."
• SW integration complexity
• HPC system using DBMS services
• HPC system providing multiuser access via Web
• Bulky data management (DBMS, RAM requirements)
• Synchronization and cooperation of different
  programming models and environments (OO
  programming, ODBMS, message passing, Web
  technologies, Java)
• Effective deployment of new SW environments
  (ENVIIDL, FastObjects)

• Anatomy of the Project

8

• System Architecture

9
System Architecture (1)

• The system architecture has been designed in
  order to meet the following goals
• fast and scalable processing performance
• application-oriented DBMS services, optimized for
  handling bulky data
• user-friendly and reliable end-to-end operations
• The final result is an integrated HW and SW
  solution, built around a Quadrics Linux Cluster
  and organized into three major subsystems
• Image Processing Subsystem
• Storage Management Subsystem
• Collection and Dissemination Subsystem

• System Architecture

10
System Architecture (2)
The Image Processing Subsystem extends the
single-node computing capabilities of ENVI (by
Research System), the most advanced remote
sensing software available on the market, to the
scalable cluster computing solution. The ENVI GUI
has been enriched with hooks and handles to
activate parallel functions, which will
dramatically speed-up the image processing
tasks. Furthermore the Storage Management
Subsystem integrates the innovative
object-oriented DBMS, FastObjects (by Poet
Software), into the input/output ENVI menu,
providing sophisticated imagery data query and
management functions. The Collection and
Dissemination Subsystem makes IMAS-D a real
end-to-end system, suitable for the automatic
collection and archiving of incoming satellite
images and for delivering via Web the processed
products.

• System Architecture

11
HW Architecture

• System Architecture

12
Real HW Picture
IMAS-D System

• System Architecture

13
System Operation Definition

• System Architecture

14
SW Actors
Legenda
IMAS-D GUI
AIPB Server(parallel proc.server)

• System Architecture

15
Data Flow between SW Modules
proc. request image data ancillary data
raw image data
processed image response info
START / STOP
proc. request image data ancillary data
processed image response info
insert/query command
search statistics retrieved image
START /STOP
query request
retrieved images
image data blocks ancillary data
processed image data blocks

• System Architecture

16
SW Module Dependencies

• System Architecture

17
Parallel ProcessingInteraction Diagram
lt------------------------- master computing node
--------------------------gt
lt-- slave computing nodes --gt
CMPI parallel programming ltslave nodegt
send image subset
4
ENVI IDL library
7 return data subset result
CMPI parallel programming ltmaster nodegt
ENVI widgets
IDL sequential code
shared mem C communication library
7 return data subset result
CMPI parallel programming ltslave nodegt
ENVI IDL library
4
send image subset

• System Architecture

18
Development tools
 
Visualization and processing
Linux Red Hat Linux distribution, installed on
QLC ENVI GUI library for remote sensing
applications IDL interpreted programming
languages for image processing C/C
compiler compilers for sequential/parallel C/C
programs MPI Message Passing Interface library,
optimized for QsNet make support tool for SW
compilation CVS Concurrent Version System, used
for SW development

• System Architecture

 
19

• Performance Results

20
Parallel Performance

• WP6 Progress Status

21
Parallel Performance

• WP6 Progress Status

22
Parallel Performance

• WP6 Progress Status

23
Parallel Performance

• WP6 Progress Status

24
Parallel Performance

• WP6 Progress Status

25

• Demo Session

26
IMASD
27
Main Menu
28
Overview of IMASD Tools
Cluster Management
Data Base Interface
Data Collection Pre-Processing Tools
Interactive use of Processing functionalities
Help
29
Cluster Management
30
Cluster Control Cluster Status

• Reserve a set of CPUs to be used for parallel
  computing

• Graphical display of CPUs status

• Check the status of the Cluster partitions

• Start the parallel server

Running 8 CPUs
Running 16 CPUs