VMPlants: Providing and Managing Virtual Machine Execution Environments for Grid Computing

1
VMPlants Providing and Managing Virtual Machine
Execution Environments for Grid Computing

• Ivan Krsul, Arijit Ganguly, Jian Zhang, José A.
  B. Fortes, Renato J. Figueiredo
• Advanced Computing and Information
  SystemsElectrical and Computer Engineering
• University of Florida

Supercomputing 2004
2
Overview

• Goal Support virtual machines (VMs) as execution
  environments for Grid computing
• Challenge Efficient on-demand instantiation of
  application-centric virtual machine environments
• Contribution Grid service providing means to
  create/configure/destroy VMs meeting application
  requirements, hiding the details of VM technology

3
Outline

• Grid Computing on Virtual Machines
• Related Issues
• Our Work
• Experimental results

4
Virtual Machines

• Whole system VMs having their own O/S, running on
  virtual hardware or instruction set
• VMware
• User-mode Linux
• Xen
• Microsoft Virtual PC

• Not Java virtual machine

5
Grid Computing on Virtual Machines

• Fundamental goal of Grid computing
• "Flexible, secure, coordinated resource sharing
  among dynamic collections of individuals,
  institutions, and resources Foster et. al
• Without forcing the Grid users to modify their
  applications to make them Grid-aware
• Key challenge to Grid middleware
• The provisioning of execution environments that
  have flexible, customizable configurations and
  allow for secure execution of untrusted code from
  Grid users

"The Anatomy of the Grid Enabling Scalable
Virtual Organizations", I. Foster, C. Kesselman,
S. Tuecke. International J. Supercomputer
Applications, 15(3), 2001
6
Virtual Machines for Grid Computing

• Resource security and user isolation
• Flexible customization and legacy support
• Site independent deployment and migration

R. Figueiredo, P. A. Dinda, J. A. B. Fortes, A
Case for Grid Computing on Virtual Machines,
Proc. International Conference on Distributed
Computing Systems (ICDCS), May 2003.
7
Problems with using physical machines
Compute Server
Compute Server
Compute Server
Grid

• Requirements
• Ch3D Linux
• ArcView Windows

Compute Server
Compute Server
Compute Server
8
Our approach Define once, instantiate on-demand
Compute Server
Compute Server
Compute Server
Grid
Middleware
Compute Server
Compute Server
Compute Server
Available at http//www.acis.ufl.edu/invigo
9
The In-VIGO virtual workspace

• Mounts user files from File Server
• Exports a VNC display
• File Manager to upload/download files
• Customizable according to user preferences

Challenge Defining such application environments
and their fast provisioning
10
Our contribution

• Directed Acyclic Graph (DAG) model for defining
  application-centric VMs
• DAG-matching technique for fast VM instantiation
• Cost-bidding for choosing compute servers for VM
  instantiation
• Grid service providing means to efficiently
  create/configure/destroy VMs, that is generic
  across VM technologies

11
Architectural Components of proposed Grid Service
VM Creation Request from Client (eg In-VIGO)
(6) VM Classad
(1) VM Request
VMShop
(2) Request Estimate
(3) VM Creation Cost
(4) Create VM
(5) VM Classad
vws010
VMPlant Daemon
vws001
VMPlant Daemon
vws005
VMPlant Daemon
vws002
vws003
VMPlant Daemon
charlie
james
bob
alice
ozzy
Host OS (VMPlant)
Host OS (VMPlant)
Host OS (VMPlant)
12
Upcoming Issues

• Specifying such application-centric VMs
• Choosing the compute servers for VM instantiation
• Fast VM creation inside the VMPlant

13
Defining application-centric VMs
VM Creation Request from Client (eg In-VIGO)
(1) VM Request
(6) VM Classad
VMShop
(2) Request Estimate
(3) VM Creation Cost
(4) Create VM
(5) VM Classad
vws010
VMPlant Daemon
vws001
VMPlant Daemon
vws005
VMPlant Daemon
vws002
vws003
VMPlant Daemon
charlie
james
bob
alice
ozzy
Host OS (VMPlant)
Host OS (VMPlant)
Host OS (VMPlant)
14
Directed Acyclic Graph (DAG) definition of a VM

• Start node - a blank machine with certain
  hardware specifications
• Each DAG node represents actions to be performed
  inside the VM guest, or on the VM host
• Actions can be arbitrary command executions,
  uploading files, installing packages etc
• Allows specifying error handling actions during
  the VM configuration process, and also attributes
  for querying the VM later

15
Example Fragment of the DAG for an In-VIGO
virtual workspace
VM Creation Request from Client (e.g. In-VIGO)
VM Request (Hardware settings and DAG describing
the VM)
VMShop
16
The XML representation of DAG for an In-VIGO
Virtual Workspace
17
Choosing compute servers for VM instantiation
VMShop Client (e.g. In-VIGO)
(6) VM Classad
(1) VM Request
VMShop
(3) VM Creation Cost
(2) Request Estimate
(4) Create VM
(5) VM Classad
vws010
VMPlant Daemon
vws001
VMPlant Daemon
vws005
VMPlant Daemon
vws002
vws003
VMPlant Daemon
charlie
james
bob
alice
ozzy
Host OS (VMPlant)
Host OS (VMPlant)
Host OS (VMPlant)
18
Cost Bidding Selecting a VMPlant
VMShop
Create 256 MB VM
charlie
VMPlant Daemon
VMPlant Daemon
VMPlant Daemon
bob
VMPlant Daemon
alice
ozzy
VMPlant Daemon
VMPlant Daemon
vws010
vws005
vws002
vws003
Host OS (VMPlant)
Host OS (VMPlant)
Host OS (VMPlant)
512 MB host 256 MB VM
1 GB host 2 256 MB VMs
512 MB host 256 MB VM
Cost Desired Guest Memory (Total Host Memory
Current total memory usage by VM guests)
19
Inside the VMPlant
vws001
VMPlant Daemon
vws002
vws003

• VM

classad

• Create

• VM

• Estimate cost

• Estimate cost

• Cost estimate

• Cost estimate

james
alice
ozzy
Host OS (VMPlant)
VMPlant
daemon

• Create

• Create

VM classad

• Estimate

• Lookup cached VM

VM Warehouse (Pre-built VM images and associated
DAGs)
Production Process Planner (DAG matching)

• VM Warehouse

• Planner (PPP)

VM classad
Library of VM images

• (

• VMware

Production Line VMware
Production Line UML
Production Line Xen
20
DAG matching Choosing the golden image
32 MB blank machine
START
A
B
C
D
Requested VM Configuration
32 MB blank machine
START
A
B
Matching Configuration in the VM warehouse
Clone
Configure
START
A
B
C
D
21
Fast VM cloning

• Copy-on-write virtual disks (GBytes) the for the
  VM images only redo-logs (a few MBytes) have to
  be copied
• Fast VM copying of VM images
• VM images in the warehouse suspended VMs
• VMware supports suspended state
• SBUML for UML snapshots
• Saves boot-up time

22
Other Components

• VMCollector destroys virtual machines actions
  specified via destruction DAG
• VMReporter queries virtual machines on
  attributes that can be user-defined
• VMArchitect assigns network addresses oversees
  creation of clusters of VMs
• Coupled with virtual networking service to make
  VM accessible in clients network
• Leverages VNET Sundarajan, Dinda, USENIX 04

23
Implementation Status

• Our prototype
• VMShop and VMPlant
• Java (11K lines of code)
• XML request/reply over sockets
• Production line
• Perl (VMware GSX, User-mode Linux)
• Configuration virtual CD-ROM ISO image, Perl
  scripts created from the DAG
• Binding between the VMShop and VMPlants is static

24
VMPlant Grid Service

• VMShop/Plant follow XML-based request/reply
  protocol
• VMShop create(), configure(), query(), destroy()
• VMPlant create(), configure(), query(),
  destroy(), estimate-cost()
• State
• VMShop list of VMPlants
• VMPlant VM classad, run-time state
• Interactions between client/VMShop and
  VMShop/VMPlant service
• Interaction between client and VM at run-time
  arbitrary (e.g. ssh)
• To-do
• Implement WSRF-compliant VMShop/Plant interfaces
• Leverage standards and interoperability of
  services

25
Experiments with VM creation

• VMPlant 8 nodes of IBM e1350 xSeries cluster
• Linux RH7.3, dual-processor, 2.4 GHz Xeon, 1.5 GB
  RAM, 18 GB SCSI disk, GSX 2.5.1 (build 5336)
• VM Warehouse NFS-mounted storage
• Dual 1.8GHz Xeon, 1 GB RAM, 500 GB SCSI RAID5
  server
• VMShop
• Server e1350 cluster node
• Client 1.8 GHz Pentium-4/512MB, Mandrake
  9.1workstation
• VMs created
• Mandrake Linux 8.1 suspended post boot stage
• Memory sizes 32 MB, 64 MB, 256 MB

26
Experiments VM creation times
start
Install Mandrake 8.1
Clone
Config MAC
End-to-end creation time for VMs with network
IP/MAC configured
Config IP
Observation VM creation times increase with the
guest memory size
27
Experiments VM cloning times
Creating VMs sequentially on the set of 8 VMPlants
Observation VM cloning times increase with
memory pressure on host
28
Related Work

• Collective, SODA and XenoCorp
• Advocate computing on virtual infrastructure, and
  its instantiation on-demand
• DAG in VMShop allows for more flexible and
  fine-grained definitions
• SODA and Collective VMs are long-lived VMShop
  VMs persist only for the lifetime of the
  application run
• Oceano and Cluster On-Demand (COD)
• On-demand provisioning, but on physical machines
• COD is based on remote-boot and use of
  configuration templates VMShop supports cloning
  and DAG matching

29
Related Work

• Dynamic virtual environments (DVEs) Grid 04
• Virtual machines as execution environments
• DVEs provide Globus-enabled environments
  VMPlants create generic VMs, clients can interact
  with VMs using mechanisms they choose

K. Keahey, K. Doering and I. Foster, From
Sandboxes to Playground Dynamic Virtual
Environments in the Grid, Proceedings of Grid
Workshop, 2004.
30
Summary

• Problem Defining and instantiating VM execution
  environments for grid computing
• Solution VMShop allows for fast VM
  instantiation customization according to the
  application requirements
• Evidence Experimental results for VM creation

31
Acknowledgments

• In-VIGO team at UFL
• http//www.acis.ufl.edu/invigo
• Dr. Peter Dinda and Virtuoso team at NWU
• http//virtuoso.cs.northwestern.edu
• NSF Middleware Initiative
• http//www.nsf-middleware.org
• NSF Research Resources
• IBM Shared University Research
• VMware

Questions?
32

• Thank You

33
Fast VM cloning across sites

• Virtual File System support
• Use of disk caching of VM state at VMPlants

block-basedcache
buffer
NFS server
kernel
proxy
WAN
file-basedcache
disk
mem
VMM
VM state
Compute server C
VM state server S
VM Plant
VM Warehouse
Ming Zhao, Jian Zhang, Renato Figueiredo,
Distributed File System Support for Virtual
Machines in Grid Computing, Proceedings of HPDC
04
34
Experimental Results VMware VM cloning
Sequential cloning time vs. parallel cloning time
for eight VMs
Sequential VM cloning times (seconds)

• GVFS greatly reduces VM cloning time, and
  achieves speed close to the local disk setup if
  temporal locality exists across clone requests
• Good performance also achieved when VMs cloned in
  parallel in a cluster

Ming Zhao, Jian Zhang, Renato Figueiredo,
Distributed File System Support for Virtual
Machines in Grid Computing, Proceedings of HPDC
04
35
VM Configuration

• XML DAG converted into a Perl script
• Perl script written to an ISO image which serves
  as virtual CD-ROM for the VM
• Configuration achieved through a pair of scripts
  one running on the VM host, and the other inside
  the VM guest for synchronization

36
Overview of service architecture
VMPlants
VMShop
Registry

• Create
• Configure

Discover
Publish

• Query

• Destroy

• Estimate

Bind
Client

• Create
• Configure

• Query

• Destroy

37
Scalability Issues

• Number of VMPlants
• VMShop has to process too many bids from VMPlants
• Solution
• Use of VMBrokers
• Distributed processing of bids

38
VMBrokers
Estimate-Cost
VMShop
VMBroker2
VMBroker1
Cost-Estimate
VMPlant4
VMPlant3
VMPlant2
VMPlant1
39
DAG matching
A
C
D
start
B
start
A
B
E
Fails subset test
start
B
C
D
Fails prefix test
Fails partial-order test
start
A
C
B
start
A
B
E
Passes all tests Golden image