Adaptive Virtual Networking For Virtual Machinebased Distributed Computing

1
Adaptive Virtual Networking For Virtual
Machine-based Distributed Computing

• Peter A. Dinda
• Prescience Lab
• Department of Computer Science
• Northwestern University
• http//plab.cs.northwestern.edu

2
People and Acknowledgements

• Students
• Ashish Gupta, Ananth Sundararaj, Alex Shoykhet,
  Jack Lange
• Collaborators
• In-Vigo project at University of Florida
• Renato Figueiredo, Jose Fortes
• Funders/Gifts
• NSF through several awards, VMWare

3
IBM xSeries virtual cluster (64 CPUs), 1 TB RAID
Interactivity Environment Cluster, CAVE (90
CPUs), 8 TB RAID
2 Distributed Optical Testbed Clusters IBM
xSeries (14-28 CPUs), 1 TB RAID
DOT clusters with optical connectivity IBM
xSeries (14-28 CPUs), 1 TB RAID Argonne,
U.Chicago, IIT, NCSA, others
Nortel Optera Metro Edge Optical Router
Distributed Optical Testbed (DOT) Private Optical
Network
Northwestern
4
Users already know how to deal with this
complexity at another level
5
Virtuoso The Dell Model
A. Shoykhet, J. Lange, and P. Dinda, Virtuoso A
System For Virtual Machine Marketplaces,
Technical Report NWU-CS-04-39, July, 2004.
R. Figueiredo, P. Dinda, J. Fortes, A Case For
Grid Computing on Virtual Machines, Proceedings
of the 23rd International Conference on
Distributed Computing Systems (ICDCS 2003)
6
The Illusion
Users LAN
VM
User
Your machines are sitting next to you.
7
Virtual Machines

• Language-oriented VMs
• Abstract interpreted machine, JIT Compiler, large
  library
• Examples UCSD p-system, Java VM, .NET VM
• Application-oriented VMs
• Redirect library calls to appropriate place
• Examples Entropia VM
• Virtual servers
• Kernel makes it appear that a group of processes
  are running on a separate instance of the kernel
  or run OS at user-level on top of itself
• Examples Ensim, Virtuozzo, UML, VServer, FreeVSD
  
• Microkernels designed to host OSes
• Xeno VM
• Virtual machine monitors (VMMs)
• Raw machine is the abstraction
• VM represented by a single image
• Examples IBMs VM, VMWare, Virtual PC/Server,
  Plex/86, SIMICS, Hypervisor, DesQView/TaskView.
  VM/386

8
Claim

• Virtual networking for VMs enables the broad
  application of dream techniques
• Adaptation
• Resource reservation
• using existing, unmodified applications and
  operating systems
• So actual people can use the techniques

9
Questions

• Is there enough application information?
• Resource demands
• Goals
• Is there enough resource information?
• Cycles
• Bandwidth
• Are there sufficient adaptation and reservation
  mechanisms?
• Is the control loop fast enough?

10
Outline

• Motivation and claims
• VNET A virtual network for virtual machines
• And what it enables
• VTTIF Application topology inference
• Dynamic topology adaptation
• Combining VNET and VTTIF
• Current directions
• Conclusions

11
Why Virtual Networking?

• A machine is suddenly plugged into your network.
  What happens?
• Does it get an IP address?
• Is it a routeable address?
• Does firewall let its traffic through?
• To any port?

How do we make virtual machine hostileenvironment
s as friendly as the users LAN?
12
VNET A Layer 2 Virtual Network for the Users
Virtual Machines

• Why Layer 2?
• Protocol agnostic
• Mobility
• Simple to understand
• Ubiquity of Ethernet on end-systems
• What about scaling?
• Number of VMs limited (1024/user)
• Hierarchical routing possible because MAC
  addresses can be assigned hierarchically

A. Sundararaj, P. Dinda, Towards Virtual Networks
for Virtual Machine Grid Computing, USENIX VM 2004
13
A Simple Layer 2 Virtual Network
Client
Server
VM monitor
SSH
Remote VM
Virtual NIC
Physical NIC
Physical NIC
Hostile Remote Network
Friendly Local Network
14
A Simple Layer 2 Virtual Network
Client
Server
VM monitor
SSH
Remote VM
Virtual NIC
Physical NIC
Physical NIC
Hostile Remote Network
Friendly Local Network
15
A Simple Layer 2 Virtual Network
Client
Server
UDP, TCP, TCP/SSL, or SSH tunnel
VM monitor
vnetd
vnetd
Remote VM
Virtual NIC
Physical NIC
Physical NIC
Hostile Remote Network
Friendly Local Network
16
More Details
VM
Host Only Network
ethz
eth0
ethy
ethx
eth0
vmnet0
Client LAN
VNET
VNET
IP Network
Ethernet Packet Injected Directly into VM
interface
Host
Proxy
Client
Ethernet Packet Tunneled over TCP/SSL Connection
Ethernet Packet Captured by Promiscuous Packet
Filter
A collection of such Proxy/Host connections forms
a star network centered at the Proxy on the
users network
VNET 0.9 available from http//virtuoso.cs.northw
estern.edu
17
Initial Performance Results (LAN)
Faster than NAT approach Lots of room for
improvement This version you can download and
use right now
18
VNET 1.0 Bootstrapping the Virtual Network
Host VNETd
Proxy VNETd
VM

• Star topology always possible
• Topology may change
• Links can be added or removed on demand
• Virtual machines can migrate
• Forwarding rules can change
• Forwarding rules can be added or removed on
  demand

19
Current Status Snapshots
Pseudo proxy
20
VNET 1.0 Performance

• BW and latency similar to VNET 0.9
• Add/Delete Link 21 ms
• Add/Delete Rule 16 ms
• IBM e1350 cluster, 100 mbit switch

21
VNET 1.0 Topology Manipulation(Eight VMs)
Setup
Teardown
22
VNET 1.0 Topology Manipulation (Eight VMs)
23
VNET 1.0 Topology Manipulation (Eight VMs)
24
Outline

• Motivation and claims
• VNET A virtual network for virtual machines
• And what it enables
• VTTIF Application topology inference
• Dynamic topology adaptation
• Combining VNET and VTTIF
• Current directions
• Conclusions

25
VM Layer
Vnetd Layer
Physical Layer
26
Application communication topology and traffic
load application processor load
VM Layer
Vnetd Layer
Physical Layer
27
Application communication topology and traffic
load application processor load
VM Layer
Vnetd Layer
Network bandwidth and latency sometimes topology
Physical Layer
28
Application communication topology and traffic
load application processor load
VM Layer
Vnetd layer can collect all this information as
a sideeffect of packet transfers
Vnetd Layer
Network bandwidth and latency, sometimes
topology host load
Physical Layer
29
Application communication topology and traffic
load application processor load
VM Layer
Vnetd layer can collect all this information as
a sideeffect of packet transfers and invisibly
act
Vnetd Layer
Network bandwidth and latency sometimes topology
Physical Layer
30
Application communication topology and traffic
load application processor load
VM Layer

• Vnetd layer can collect all this information as
  a sideeffect of packet transfers
• and invisibly act
• VM Migration

Vnetd Layer
Network bandwidth and latency sometimes topology
Physical Layer
31
Application communication topology and traffic
load application processor load
VM Layer

• Vnetd layer can collect all this information as
  a sideeffect of packet transfers
• and invisibly act
• VM Migration
• Topology change

Vnetd Layer
Network bandwidth and latency sometimes topology
Physical Layer
32
Application communication topology and traffic
load application processor load
VM Layer

• Vnetd layer can collect all this information as
  a sideeffect of packet transfers
• and invisibly act
• VM Migration
• Topology change
• Routing change

Vnetd Layer
Network bandwidth and latency sometimes topology
Physical Layer
33
Application communication topology and traffic
load application processor load
VM Layer

• Vnetd layer can collect all this information as
  a sideeffect of packet transfers
• and invisibly act
• VM Migration
• Topology change
• Routing change
• Reservation

Vnetd Layer
Network bandwidth and latency sometimes topology
Physical Layer
34
Outline

• Motivation and claims
• VNET A virtual network for virtual machines
• And what it enables
• VTTIF Application topology inference
• Dynamic topology adaptation
• Combining VNET and VTTIF
• Current directions
• Conclusions

35
VTTIF Application Traffic Load Measurement and
Topology Inference

• Parallel and distributed applications display
  particular communication patterns on particular
  topologies
• Intensity of communication can also vary from
  node to node or time to time.
• Combined representation Traffic Load Matrix
• VNET already sees every packet sent or received
  by a VM
• Can we use this information to compute a global
  traffic load matrix?
• Can we eliminate irrelevant communication from
  matrix to get at application topology?

36
Traffic Monitoring and Reduction
VM
Host Only Network
Ethernet Packet Format SRCDESTTYPEDATA
(size) VMTrafficMatrixSRCDESTsize Each VM
on the host contributes a row and column to the
VM traffic matrix Global reduction to find
overall matrix, broadcast back to VNETs Each
VNET daemon has a view of the global network
load
ethz
eth0
vmnet0
VNET
Host
Packets observed here
37
Denoising The Matrix

• Throw away irrelevant communication
• ARPs, DNS, ssh, etc.
• Find maximum entry, a
• Eliminate all entries below aa
• Very simple, but seems to work very well for BSP
  parallel applications
• Remains to be seen how general it is

38
Offline Results Synthetic Benchmark
39
NAS IS Benchmark
40
NAS IS Benchmark
41
Online Challenges

• When to start? When to stop?
• Traffic matrix may not be stationary!
• Synchronized monitoring
• All must start and stop together

42
When To Start? When to Stop?
Reactive Mechanisms
Proactive Mechanisms
Start when traffic rate exceeds threshold Stop
when traffic rate exceeds a second
threshold Non-uniform discrete event sampling
Provide support for queries by external agent
Keep multiple copies of the matrix, one for
each resolution (1s, 2s, 4s, etc)
What is the Traffic Matrix from the last time
there was at least one high rate source?
What is the Traffic Matrix for the last n seconds
?
43
Overheads (100 mbit LAN)

• Essentially zero latency impact
• 4.2 throughput reduction versus VNET

A. Gupta, P. Dinda, Inferring the Topology and
Traffic Load of Parallel Programs Running In a
Virtual Machine Environment, JSSPP 2004.
44
Online NAS IS on 4 VMs
45
Outline

• Motivation and claims
• VNET A virtual network for virtual machines
• And what it enables
• VTTIF Application topology inference
• Dynamic topology adaptation
• Combining VNET and VTTIF
• Current directions
• Conclusions

46
Dynamic Topology Adaptation

• VTTIF reactive mechanism run continuously
• On topology change, adjust VNET topology, adding
  links in priority order
• Corresponding forwarding rules also added
• Measure performance (running time) of application
  (BSP patterns application)

A. Sundararaj, A. Gupta, P. Dinda, Dynamic
Topology Adaptation in a Virtual Network of
Virtual Machines, In Submission
47
Inference and Adaptation (8 VMs, LAN)
48
Example Result (all-to-all, 8 VMs, LAN)
49
Outline

• Motivation and claims
• VNET A virtual network for virtual machines
• And what it enables
• VTTIF Application topology inference
• Dynamic topology adaptation
• Combining VNET and VTTIF
• Current directions
• Conclusions

50
Adaptation With Migration

• Learn how to adapt using Virtuosos VM migration
  capabilities
• Virtuoso migration times with rsync
• 300 seconds (1.1 GB machine)
• 50 seconds (100 MB machine)
• Versioning file system approaches
• Data point CMU ISR project 2.5-30 seconds for
  personal windows VM

51
Policy Avoidance Routing

• Multi-site collaborations often stymied by
  interactions between per-site network security
  policies
• VNET opportunity find a path on behalf of
  application where one exists, but is obscured
• Example NAT Traversal
• RFC 3489 / STUN (chownat)
• Example Tunneling through initiation protocol
• HTTP or SSH

52
Physical Network Measurement

• Use existing application traffic to measure
  underlying physical network
• Passive packet dispersion techniques
• With Bruce Lowekamp, WM
• Topology inference
• With Bruce Lowekamp, WM

M. Zangrilli and B. Lowekamp, Using Passive
Traces of Application Traffic in a Network
Monitoring System, HPDC 2004.
53
Integration With Resource Prediction
Visit rps.cs.northwestern.edu for more info and
downloads
54
Improving the Backbone

• Replacing the proxy star with a multisource
  muliticast system for higher performance and
  resilience
• FatNemo protocol
• Arrange nodes into fat tree

S. Birrer, D. Lu, F. Bustamante, Y. Qiao, P.
Dinda, FatNemo Building a Resilient Multi-Source
Multicast Fat-Tree, WCCD 2004
55
Extended Application Inference

• Offered computational load
• VM-internal performance data
• Synchronization points and waiting
• Inference of application goals
• Simple layered API for getting more application
  information into system

56
Making the Fast Path Fast

• Move VNET forwarder into kernel of host OS
• Guest OS device driver to directly communicate
  out of VM to VNET Forwarder
• Inference may make deposit message passing
  possible
• Goal Minimal overhead BW and latency for using
  VNET, even on gigabit and faster networks

57
Leveraging Optical Networking

• Use inferred application topology to do light
  path setup on behalf of application
• Currently ICAIR ODIN system, DOT network

58
Related Work

• Collective / Capsule Computing (Stanford)
• VMM, Migration/caching, Hierarchical image files,
  Attestation
• Internet Suspend/Resume (CMU/Intel)
• Your VM follows you around (will be deployed on
  CMU campus)
• Denali (U. Washington)
• Highly scalable VMMs (1000s of VMMs per node)
• CoVirt (U. Michigan)
• Xenoserver (Cambridge)
• SODA (Purdue)
• Virtual Server, fast deployment of services
• Ensim
• Virtual Server, widely used for web site hosting
• WFQ-based resource control released into
  open-source Linux kernel
• Virtouzzo (SWSoft)
• Ensim competitor
• Available VMMs IBMs VM, VMWare, Virtual
  PC/Server, Plex/86, SIMICS, Hypervisor,
  DesQView/TaskView. VM/386

59
Conclusions

• Virtual machines on virtual networks as the
  abstraction for distributed computing
• Virtual network as a fundamental layer for
  measurement and adaptation
• Status
• Virtuoso prototype running on our cluster
• VNET 0.9 released.
• VNET 1.0 (with VTTIF) in progress
• Wayback versioning file system released

60
For MoreInformation

• Prescience Lab
• http//plab.cs.northwestern.edu
• Virtuoso
• http//virtuoso.cs.northwestern.edu
• Join our user comfort study!
• http//comfort.cs.northwestern.edu
• Join our intrusion detection study!
• http//ga-ids.cs.northwestern.edu