OWLBased Resource Discovery for InterCluster Resource Borrowing

1
OWL-Based Resource Discovery for Inter-Cluster
Resource Borrowing

• Hideki Yoshida
• Toshiba Corporation
• Corporate RD Center

2
Outline

• Target systems and problem
• Conventional grid vs. Inter-Cluster Resource
  Borrowing
• Resource discovery using OWL
• Implementation

3
Target systems and problem
4
Target of this research

• Cluster systems for business applications like
• Web application server
• Customer relationship management (CRM)
• Similar to Commercial Datacenter Scenario in
  OGSA Use Cases
• Mainly computational grid, but not scientific
  computation

5
Problem in business application cluster systems

• High load fluctuation according to day and time
• ?Resource adjustment within a cluster is not
  sufficient!
• How can we realize multi-cluster resource
  adjustment?

6
Conventional grid vs. Inter-Cluster Resource
Borrowing
7
Job execution in current clusters

• Cluster Manager controls job execution
• Failover is done reliably in case of fault
• Resource allocation within a cluster is managed
  centrally
• Resource allocation policy to jobs are reflected

Cluster Manager
Cluster
Node
Job
Job
8
Conventional gridJobs are moved

• Execution of jobs are delegated to other clusters
• Discrepancy with job management in current
  cluster systems

• Does Cluster B really execute my job?
• Does B consider my resource allocation policy?

Cluster A
Cluster B
Cluster Manager
Cluster Manager
Node
Node
Node
Node
Job
Job
Job
9
Inter-Cluster Resource BorrowingResources are
moved

• Jobs are moved only within a cluster
• Borrow nodes between clusters in case of shortage

Sharing of Resource Information
Resource Request
Cluster
Borrow a node from another cluster
Cluster Manager
Cluster Manager
Node
Node
Node
Node
Job
Job
Job
Job
10
But how can we move a node between clusters?

• Node transfer can be implemented at various
  levels
• Application level
• Virtual machine level
• Network boot level
• Currently we are investigating SAN-boot
• Boot image is on a disk connected to a
  Storage-Area-Network
• Already implemented by our in-house cluster
  manager

11
Node transfer using SAN-boot

• Borrowing cluster has a spare boot image
• Borrowed node is rebooted using this image
• SAN is used to change boot disk

Cluster
Cluster Manager
Cluster Manager
Borrow a node from other cluster
Node
SAN
Change node-disk mapping using SAN
Boot Image
Boot Image
Spare Boot Image
12
Summary of difference

• Conventional grid
• Each resource is managed by a particular cluster
• Jobs are moved
• Inter-Cluster Resource Borrowing (our proposal)
• Each job is managed by a particular cluster
• Resources are moved
• ?Same level of job management

13
Resource discovery using OWL
14
Problem Discovery of heterogeneous resources

• Resources needs description for discovery
• Ever increasing diversity of available/required
  resources
• Multiple clusters (with their own goals)
• Multiple applications
• Multiple operating systems
• Multiple processor architectures
• ?Fixed resource description specification is not
  sufficient!

15
Solution Resource discovery using OWL

• Resource models can be provided at run-time
• Extensible description format
• Off-the-shelf inference engines can be used for
  matching of descriptions
• Currently we are using Jena2

16
Resource descriptions

• Three kinds of OWL descriptions are used for
  resource discovery

A cluster has physical nodes. Each node is of
some architecture.
Model Description
Cluster1 has a physical node called
Node1. Node1 is of x86 architecture.
I want a physical node of x86 architecture.
Instance Description
Requirement Description
Inference Engine
Cluster1 is likely to have that resource.
Search Result
17
Descriptions specific to Inter-Cluster Resource
Borrowing

• In addition to descriptions common to grid
  resources, some exotic descriptions are needed
• Descriptions related to node transfer
• Available layers of node transfer
• e.g. SAN-boot
• Possible range of node transfer
• e.g. SAN topology
• Policies related to node transfer

18
Implementation
19
Overall system structure
Grid Manager
Resource Description
Resource Description
OGSI-Based Protocol
Grid Layer
Grid Agent
Grid Agent
Resource Information
Node Request
Resource Information
Node Request
Node Transfer
Cluster Manager
Cluster Manager
Node
Node
Node
20
Cluster Layer

• Execution management of jobs
• Resource management/allocation within a cluster
• Not service-based
• Applications can run without knowing about
  clusters or grids
• with external requirement description files

21
Grid Layer

• Grid Manager Grid Agent
• Global adjustment of resource allocation
• Sharing of resource information
• A kind of information service
• Processing of resource request
• Resolution of resource requirements
• Determination of candidate cluster to provide
  resource
• Node transfer
• Service-based (Globus 3.2)

22
Two-phase resource discovery

• Candidate cluster discovery in Grid Manager
• Filtering using pre-registered resource
  information
• Resource discovery in Grid Agent
• Find actual resource considering additional
  dynamic resource information

23
Current status

• Stand-alone prototype of Grid Layer is
  implemented
• Resource discovery using requirement description
• Resource request
• Node transfer is just simulated
• Future directions
• Interface to real cluster managers
• ?Real node transfer
• Aggregation of resource information

24
Summary

• Inter-Cluster Resource Borrowing for
  multi-cluster resource adjustment
• Flexible resource discovery using OWL
• Current and planned implementation

25
Topics we want to know

• Other efforts toward low-level resource
  allocation using semantic description
• OWL or RDF version of CIM schema?
• Ways to search the best resource
• Boolean matching is straightforward
• Can the fastest server that can run this
  application be found using OWL or RDF?

26
Thanks