The MEDIGRID Distributed Framework for Multi-risk Assessment of Natural Hazards

1
The MEDIGRID Distributed Framework for Multi-risk
Assessment of Natural Hazards

• Isabella Bovolo1, Simon Abele1, James Bathurst1
  , David Caballero2, Marek Ciglan3, Branislav Simo3

1University of Newcastle upon Tyne, UK 2Tecnoma,
Spain, 3Institute of Informatics, Slovakia
GOCE-CT-2003-004044
2
(No Transcript)
3
Outline

• Introduction
• MEDIGRID System
• Example application
• Conclusion

4
Outline

• Introduction
• MEDIGRID System
• Example application
• Conclusion

5
Mediterranean Grid of Multi-risk Data and Models

• Project Aim
• Integrate several, independent, pre-existing,
  natural hazard models into one single system.
• Run any of the models in sequence using data
  stored remotely
• ? multi-model, multi-hazard assessment tool

6
Challenges

• Models
• developed in isolation
• different software architectures
• different data needs
• applicable to different spatial / temporal scales
• Identification of common data themes and links
  between models
• How do models communicate? Common data formats
  for data exchange between models

7
Models
Hydrology, Erosion, Landslide Model
FMIS FIRESTATION Wind field /FMC / ROS Fire
Spead Engine
SHETRAN
Flood Model
Forest Fire Model
ALADIN,MM5,HSPF,NLC, DAVEF Meteorology,
Hydrology and Hydraulics
SPREAD
PROMETHEUS
Vegetation Recovery Model
Erosion Model
8
Models and Data Flows
SHETRAN Hydrology, Erosion, Landslide Model
Forest Fire Model
Vegetation Recovery Model
9
Test sites

• Western Spain
• Southern France
• Central Portugal

10
Outline

• Introduction
• MEDIGRID System
• Example application
• Conclusion

11
MEDIGRID System

• Provide modular decision support framework for
  assessing multiple hazards based on Grid-enabled
  applications and distributed architecture.

12
Globus Toolkit

• Globus Toolkit environment installed on all
  MEDIGRID process data storage nodes
• Common grid technology
• Secure infrastructure
• Allows data / computational resources to be
  deployed via internet
• Allow users to access the system remotely through
  a web portal (containing nportlets)

13
MEDIGRID System
1. User Interface
6
2. System Services
5
3. User Services
3
2
4
1
14
1. User Interface

• GridSphere portlet container
• Open-source portlet framework
• Offers common Grid portal features /
  functionality
• Provides shared access point to Grid services
  resources

• Allows
• Single sign-on
• Credential management
• Job submission / monitoring
• Resources management
• Models
• Model connectivities
• Data

15
2. System Services

• Issues
• Variations in model data security / access/use
  policies across partner sites
• Input data must be stored, maintained, accessed,
  be searchable
• Inputs / Results should be a single data set
• Current Grid Middleware have not implemented
  multi-platform support for WS- and Grid
  standards e.g. GridFTP, GRAM only work on
  Linux platform

16
2. System Services

• Several customised components developed
• Data services
• Data Transfer Service
• Data Resource Management
• Job submission service
• Implemented with Globus Java WSRF Core
• Industry standard
• Link between Web service and Grid technologies

17
Data services architecture
User
Portal
Data transfer WS interface
WS interface
WS interface
Auth Authz
Data Transfer
Medigrid
Replica Location Service
Metadata Catalog
GridFTP
18
Data Access policy

• Data security
• Grid Certificates signed by Medigrid
  Certificate Authority (CA)
• Hosts (i.e. computational/data nodes)
• Individual users
• Access Control Lists (ACL) define what resources
  hosts users can access

19
Data services architecture
User
Portal
Data transfer WS interface
WS interface
WS interface
Auth Authz
Data Transfer
Medigrid
Replica Location Service
Metadata Catalog
GridFTP
20
Data Management

• Data Transfer Service (DTS)
• Custom MEDIGRID Grid service
• Extends GridFTP
• Provides secure data transfer mechanism, applies
  data access / security policies
• Data Transfer Operations
• Upload data to Medigrid sites
• Download data from Medigrid sites
• Third-party transfer between Medigrid sites

21
Data Management

• Data Transfer Service, cont
• Integrated with
• Replica Location Servce (RLS)
• (Globus Toolkit)
• Metadata catalog Service (MCS)
• (GriPhyN / NVO project component)

22
Data services architecture
User
Portal
Data transfer WS interface
WS interface
WS interface
Auth Authz
Data Transfer
Medigrid
Replica Location Service
Metadata Catalog
GridFTP
23
Metadata Catalog Service

• MCS components
• set of standard interfaces and APIs for storing
  and accessing resource metadata
• query mechanisms for accessing the database
• allows multiple metadata categories to be
  associated with any logical file name
• domain-independent metadata attributes
• extensibility for user-defined attributes

24
Data Services
25
Virtual Filesystem Browser
File / Directory View Copy Move Delete Create
26
Metadata attributes portlet View metadata
attributes Edit metadata attributes Add metadata
attributes
27

• Metadata search portlet
• metadata search based on attributes values
• specify multiple query conditions

28
Job Submission Service

• WSRF Web Service client tools
• Runs on Windows and Linux
• Acts as an interface to a defined, fixed
  executable
• Executable is configurable via user supplied
  parameters, however,
• does not allow specification of new .exe files
• Monitors job state
• Multiple jobs to computational services can be
  submitted and managed.
• Jobs are queued and executed one at a time

29
3. User Services

• Computational nodes
• Host hazard assessment applications
• Simple complex

30
Model adaptations

• Modify code to be standalone executable
• Remove graphical user interface
• command line arguments
• Convert to web-service to run remotely over the
  internet
• Develop pre- and post-processing tools to
  translate data to and from a common format (Grass
  ASCII)

31
3. User Services

• Through portal (containing portlets), users can
• Run models and
• Specify model interconnectivities
• Choose pre-defined configuration input files
• Customise parameters
• Select outputs required
• Download / View results

32
MEDIGRID System ExampleWorkFlow
DataRes1
DataRes1
file_1 file_2 file_3
file_1 file_2 file_3
FFireRes
FFireRes
Site 1
Site 2
ffire_data_1 ffire_data_2 ffire_data_3
ffire_data_4 ffire_data_5 ffire_data_6
Medigrid Central Data Services
RLS
DataRes1
file_1 file_2 file_3
Metadata Catalog
LSlidesRes
Site 3
lslidesdata_1 lslidesdata_2 lslidesdata_3
33
DataRes1
DataRes1
file_1 file_2 file_3
file_1 file_2 file_3
FFireRes
FFireRes
Site 1
Site 2
ffire_data_1 ffire_data_2 ffire_data_3
ffire_data_4 ffire_data_5 ffire_data_6
Medigrid Central Data Services
RLS
User
DataRes1
file_1 file_2 file_3
Metadata Catalog
LSlidesRes
Site 3
lslidesdata_1 lslidesdata_2 lslidesdata_3
34
DataRes1
DataRes1
file_1 file_2 file_3
file_1 file_2 file_3
FFireRes
FFireRes
Site 1
Site 2
ffire_data_1 ffire_data_2 ffire_data_3
ffire_data_4 ffire_data_5 ffire_data_6
Medigrid Central Data Services
RLS
User
DataRes1
file_1 file_2 file_3
Search Metadata
Metadata Catalog
LSlidesRes
Site 3
lslidesdata_1 lslidesdata_2 lslidesdata_3
35
DataRes1
DataRes1
file_1 file_2 file_3
file_1 file_2 file_3
FFireRes
FFireRes
Site 1
Site 2
ffire_data_1 ffire_data_2 ffire_data_3
ffire_data_4 ffire_data_5 ffire_data_6
Medigrid Central Data Services
RLS
User
DataRes1
file_1 file_2 file_3
User finds LFNs of required data
Metadata Catalog
LSlidesRes
Site 3
lslidesdata_1 lslidesdata_2 lslidesdata_3
36
DataRes1
DataRes1
file_1 file_2 file_3
file_1 file_2 file_3
Submit Job, specify LFNs of data sets (e.g.
ffire_data_1)
FFireRes
FFireRes
Site 1
Site 2
ffire_data_1 ffire_data_2 ffire_data_3
ffire_data_4 ffire_data_5 ffire_data_6
Medigrid Central Data Services
RLS
User
DataRes1
file_1 file_2 file_3
Metadata Catalog
LSlidesRes
Site 3
lslidesdata_1 lslidesdata_2 lslidesdata_3
37
DataRes1
DataRes1
file_1 file_2 file_3
file_1 file_2 file_3
Locate ffire_data_1
FFireRes
FFireRes
Site 1
Site 2
ffire_data_1 ffire_data_2 ffire_data_3
ffire_data_4 ffire_data_5 ffire_data_6
Medigrid Central Data Services
RLS
User
DataRes1
file_1 file_2 file_3
Metadata Catalog
LSlidesRes
Site 3
lslidesdata_1 lslidesdata_2 lslidesdata_3
38
DataRes1
DataRes1
file_1 file_2 file_3
file_1 file_2 file_3
ffire_data_1 Site 1
FFireRes
FFireRes
Site 1
Site 2
ffire_data_1 ffire_data_2 ffire_data_3
ffire_data_4 ffire_data_5 ffire_data_6
Medigrid Central Data Services
RLS
User
DataRes1
file_1 file_2 file_3
Metadata Catalog
LSlidesRes
Site 3
lslidesdata_1 lslidesdata_2 lslidesdata_3
39
DataRes1
DataRes1
file_1 file_2 file_3
file_1 file_2 file_3
Perform authz
FFireRes
FFireRes
Site 1
Site 2
ffire_data_1 ffire_data_2 ffire_data_3
ffire_data_4 ffire_data_5 ffire_data_6
Medigrid Central Data Services
RLS
User
DataRes1
file_1 file_2 file_3
Metadata Catalog
LSlidesRes
Site 3
lslidesdata_1 lslidesdata_2 lslidesdata_3
40
DataRes1
DataRes1
file_1 file_2 file_3
file_1 file_2 file_3
Download data ffire_data_1
FFireRes
FFireRes
Site 1
Site 2
ffire_data_1 ffire_data_2 ffire_data_3
ffire_data_4 ffire_data_5 ffire_data_6
Medigrid Central Data Services
RLS
User
DataRes1
file_1 file_2 file_3
Metadata Catalog
LSlidesRes
Site 3
lslidesdata_1 lslidesdata_2 lslidesdata_3
41
DataRes1
DataRes1
file_1 file_2 file_3
file_1 file_2 file_3
FFireRes
FFireRes
Site 1
Site 2
ffire_data_1 ffire_data_2 ffire_data_3
ffire_data_4 ffire_data_5 ffire_data_6 ffire_data_
1
Medigrid Central Data Services
RLS
User
DataRes1
file_1 file_2 file_3
Metadata Catalog
LSlidesRes
Site 3
lslidesdata_1 lslidesdata_2 lslidesdata_3
42
DataRes1
DataRes1
file_1 file_2 file_3
file_1 file_2 file_3
Register data file replica
FFireRes
FFireRes
Site 1
Site 2
ffire_data_1 ffire_data_2 ffire_data_3
ffire_data_4 ffire_data_5 ffire_data_6 ffire_data_
1
Medigrid Central Data Services
RLS
User
DataRes1
file_1 file_2 file_3
Metadata Catalog
LSlidesRes
Site 3
lslidesdata_1 lslidesdata_2 lslidesdata_3
43
DataRes1
DataRes1
Run Job
file_1 file_2 file_3
file_1 file_2 file_3
FFireRes
FFireRes
Site 1
Site 2
ffire_data_1 ffire_data_2 ffire_data_3
ffire_data_4 ffire_data_5 ffire_data_6 ffire_data_
1
Medigrid Central Data Services
RLS
User
DataRes1
file_1 file_2 file_3
Metadata Catalog
LSlidesRes
Site 3
lslidesdata_1 lslidesdata_2 lslidesdata_3
44
DataRes1
DataRes1
Job Finished
file_1 file_2 file_3
file_1 file_2 file_3
FFireRes
FFireRes
Site 1
Site 2
ffire_data_1 ffire_data_2 ffire_data_3
ffire_data_4 ffire_data_5 ffire_data_6 ffire_data_
1 job_output_1
Medigrid Central Data Services
RLS
User
DataRes1
file_1 file_2 file_3
Metadata Catalog
LSlidesRes
Site 3
lslidesdata_1 lslidesdata_2 lslidesdata_3
45
DataRes1
DataRes1
Job Finished
file_1 file_2 file_3
file_1 file_2 file_3
Register new data file (job_output_1)
FFireRes
FFireRes
Site 1
Site 2
ffire_data_1 ffire_data_2 ffire_data_3
ffire_data_4 ffire_data_5 ffire_data_6 ffire_data_
1 job_output_1
Medigrid Central Data Services
RLS
User
DataRes1
file_1 file_2 file_3
Metadata Catalog
LSlidesRes
Site 3
lslidesdata_1 lslidesdata_2 lslidesdata_3
46
DataRes1
DataRes1
file_1 file_2 file_3
file_1 file_2 file_3
Job Finished
FFireRes
FFireRes
Site 1
Site 2
ffire_data_1 ffire_data_2 ffire_data_3
ffire_data_4 ffire_data_5 ffire_data_6 ffire_data_
1 job_output_1
Medigrid Central Data Services
RLS
User
DataRes1
file_1 file_2 file_3
Metadata Catalog
LSlidesRes
Site 3
lslidesdata_1 lslidesdata_2 lslidesdata_3
47
DataRes1
DataRes1
file_1 file_2 file_3
file_1 file_2 file_3
FFireRes
FFireRes
Site 1
Site 2
ffire_data_1 ffire_data_2 ffire_data_3
ffire_data_4 ffire_data_5 ffire_data_6 ffire_data_
1 job_output_1
Job Finished
Medigrid Central Data Services
RLS
User
DataRes1
file_1 file_2 file_3
Metadata Catalog
LSlidesRes
Site 3
lslidesdata_1 lslidesdata_2 lslidesdata_3
48
Visualisation

• Users can
• download files text / HDF
• 1D graphs
• 2D maps, (GRASS GIS -gt png, jpg)
• 3D pictures / virtual env. (Blender/VRML)

49
Outline

• Introduction
• MEDIGRID System
• Example application
• Conclusion

50
Case Study Area

• Extremadura
• Semi-arid, Med climate
• lt600 mm year irregular rainfall
• -ve hydrological balance in area (aquifers not
  guaranteed to totally recharge)
• Depends on large nearby rivers

51
Valencia de Alcántara (02/08/2003)

• The fire
• Started by lightning in Portugal
• Crossed Spanish border the 03/08/03
• Burned gt 13,000 ha in one week
• Mostly pine forest, livestock and agricuture
• Effect on vegetation and catchment hydrology

52
(No Transcript)
53
Fire area
Spain - Valencia de Alcántara
02/08/2003
13,692 ha burned
30 km
54
Fire Spread Engine

• Semi-empirical model
• Estimates surface fire propagation as function of
  time
• discrete process of ignitions across a regular
  grid using cellular-automata algorithms
• Driven by surface fuels only

55
Fire Simulations

• Simulations using different ignition points
• 20 simulation hours (14s realtime)
• Time interval 60 minutes
• 3 dead fuel moisture level
• 60 live fuel moisture elevel
• 340 wind vector azimuth
• 22 km / hr windspeed

56
(No Transcript)
57
SHETRAN model
Physically based, spatially distributed,
catchment hydrology and sediment transport
modelling system capable of simulating

• land phase of the hydrological cycle
• soil erosion and sediment yield arising from
  raindrop impact and overland flow and shallow
  landsliding
• models catchment up to 500 km2 using grids of
  100 m to 2000 m

SHETRAN V4
Evapotranspiration
Landslides, erosion sediment transport
Canopy interception
Overland channel flow
Snowmelt
Root zone
3D Variably saturated subsurface flow model
Stream aquifer interactions
58
SHETRAN simulations
59
Results

• Maps of burned areas
• Hydrology
• Sediment Discharge
• Erosion maps
• Landslide maps
• Aids
• Pre- and Post-fire management

60
Outline

• Introduction
• MEDIGRID System
• Example application
• Conclusion

61
Conclusion

• Prototype MEDIGRID system testbed provides a
  decision support framework for fire and other
  natural hazards impact assessment
• Early real world example of large scale Env. Grid
  Computing system
• Provides concepts, advice, philosophy on
  challenges to others
• Suitable for use by other scientific communities

62
Conclusion

• MEDIGRID system will serve as platform for better
  collaboration between researchers / data /
  hardware / software
• Improve decision making process and outcomes of
  environmental impact assessment

63
Further Information

• s.j.abele_at_ncl.ac.uk
• http//ups.savba.sk/medigrid/
• http//www.eu-medigrid.org/
• Computers Geosciences