Gridbus Resource Broker for Application Service Costs-based Scheduling on Global Grids: A Case Study in Brain Activity Analysis

1
Gridbus Resource Broker for Application Service
Costs-based Scheduling on Global Grids A Case
Study in Brain Activity Analysis

• Srikumar Venugopal1, Rajkumar Buyya1, Susumu Date2

1.Grid computing Distributed Systems (GRIDS)
Lab. The University of Melbourne
Melbourne, Australia www.gridbus.org/ 2.
Cybermedia Center, Osaka University
2
What does a Resource Broker do?

• Gets user/application requirements
• Discovers resources like computational nodes,
  data sources, etc.
• Establishes costs, user credit, etc.
• Makes decisions about the optimal schedule for
  jobs
• Dispatches jobs

Application
Accounting Services
Information Services
Resource Broker
Cataloguing Services

Grid Nodes
3
Architecture of Gridbus Scheduler
Data Catalogue
Grid Info Service
ASP Catalogue
Grid Market Directory
4
Gridbus Scheduler

• Interfaces with
• Application Development - Visual Parametric Tool
• Information Services - Grid Market Directory
  (Cost), GRIS ,etc.
• Accounting Services - Grid Trading Service,
  GridBank
• Cataloguing Services - Application Catalog,
  Replica Catalog
• Job Dispatcher
• Nimrod-G (for parametric jobs)
• Gridbus Dispatcher (for data intensive,
  reservation, P-GRADE support, etc.) work in
  progress
• Supports
• User-specified QoS parameters such as Deadline,
  budget, etc.
• Application Cost or Hardware Cost (CPU, etc)
• Cost from Grid Market Directory or Flat File
• Cost, Time or Cost-Time Optimization.

5
Application Service Costs?

• Present Approach to Processing Cost -
• Timeshare or CPU cycles used
• Users more interested in the cost of getting
  job done than amount of processing power consumed
• New Approach to Cost -
• Application Service Costs charge for using the
  application once.
• Different costs for different applications
  depends on provider
• Broker finds Cost through Grid Market Directory.

6
Scheduling Algorithms

• Gridbus Scheduler implements
• Cost Optimization
• Minimize computational cost (within deadline)
• Time Optimization
• Minimize execution time (within budget)
• Cost-time Optimization
• Similar to cost-optimization
• Implemented for first time.

7
Scheduling (contd..)

• Uses past performance to forecast each machines
  capacity
• The rate of completion is averaged to compensate
  for any spikes or troughs
• Cost Optimization
• Gives maximum jobs to the cheapest machine
• Time Optimization
• Gives jobs to machines based on consumption rate
  but limited by budget per job
• Cost-Time Optimization
• Distributes jobs among the machines of
  consumption sorted by their consumption rate

8
Cost Optimization No. of Jobs Done vs time
9
Cost-Time Optimization No. of Jobs Done vs Time
10
Time Optimization No. of Jobs Done vs Time
11
Comparison of Scheduling Algorithms

• All experiments were started with
• No of Jobs 200
• Deadline 2hrs
• Budget 600 Grid

Start Time Completion Time Budget Consumed (Grid )
Cost 1000 a.m. 1127 a.m. 188
Cost-Time 1140 a.m. 1208 p.m. 277
Time 1230 p.m. 1259 p.m. 274
12
Case Study Brain Activity Analysis

• In Collaboration with Osaka University, Japan
• Computationally and data intensive

13
MEG Data/Brain Activity Analysis

• MEG (Magnetoencephalography)
• Achieve both non-invasiveness and high degree of
  measurement accuracy
• cf. EEG (Electroencephalography), ECoG
  (Electrocorticography)
• Measure functional data on multiple points around
  the head
• Promising among medical doctors and brain
  scientists.

A
B
http//www.ctf.com
14
MEG data analysis
DV transfer
Osaka Univ.
Data Generation
Osaka Univ. Hospital
Data Analysis
Life-electronics laboratory, AIST
Cybermedia Center

• Provision of MEG
• Provision of expertise in
• the analysis of brain function

15
MEG data analysis
16
Requirements

• Computational and data intensive problem
• The number of MEG instruments available is small.
• Knowledge of scientists is distributed.
• No database?
• Different group uses different analysis methods
  for different data..
• Many medical institutions and hospitals have no
  computers and that can satisfy doctors analysis
  demand.

17
Wavelet cross-correlation analysis
Sensor A
Sensor B
Raw MEG Data
This image indicates that a brain signal with
frequency f was detected earlier in Sensor B
than in Sensor A.

• This analysis procedure needs to be performed for
  each pair of MEG sensors. E.g. 64ch -gt 2016

18
New Approach Users QoS Requirements driven MEG
Data Analysis on the Grid
Analysis All pairs (64x64) of MEG data by
shifting the temporal region of MEG data over
time 0 to 29750 64x64x29750 jobs

• Provision of MEG analysis

19
Grid Enabling MEG data analysis

• Nature
• fine-grained jobs
• small data sets
• Data Sets on Source Node
• High Latency for small jobs
• Lower Efficiency
• Hence, data sets were replicated on each node
• Application changed to access local datasets
• ./metameg-datapath time_offset time_offset_step
  meg_sensors_count Meg_data_path
• Output is collated at the source node and then
  visualized
• Grid Enabled in very short time 1 week

20
Conclusion

• Introduced Gridbus Resource Broker using
  Application Service Cost
• Described the Scheduling Algorithms followed
• Presented Case Study of Brain Activity Analysis
  using our Resource Broker
• Future Work
• Integration with Accounting Mechanisms such as
  GridBank
• Support for Group Scheduling and Economic-based
  Advance Reservation of Resources