Anemone: network sampling in space not time

1
Anemonenetwork sampling in space not time

• R Mortier, R Isaacs, A Donnelly, E Cooke, P
  Barham
• Systems and Networking

2
Network management

• Monitoring and controlling large complex networks
• The network is slow
• Why? Network or services? Globally or locally?
• Answering these questions is a complex and
  expensive process
• IP uses highly distributed protocols
• Current tools all consider local information
• Its getting worse
• Opaque (encrypted, tunnelled) traffic is
  increasing
• Routers already highly complex and hard to modify
• Application traffic patterns are no longer simple
  
• e.g. email is no longer just SMTP

3
Anemone

• An endsystem network management platform
• Collect flow information from endsystems, and
• Combine with topology information from routeing
  protocols
• Endsystems have more information about their
  traffic than any network device
• No router support required
• A platform to support many applications
• Currently concentrating on managed networks
• E.g. governments, enterprises, etc
• High complexity, high value
• High degree of endsystem control

4
Anemone
management application queries
Anemone platformresponses
(4)
Anemoneplatform
(3)
flowdata
flowdata
(1)
(1)
end-systemmonitoring
topology data
flow detector
flow detector
flow detector
flow detector
flow detector
flow detector
flow detector
flow detector
flow detector
flow detector
flow detector
flow detector
flow detector
flow detector
(2)
packet sniffer
packet sniffer
packet sniffer
packet sniffer
packet sniffer
packet sniffer
packet sniffer
IP network
5
Applications

• Real-time and historical analysis
• Current topology ingress, egress flows Big
  Picture
• Capacity planning, anomaly detection
• Modelling what if scenarios
• Plug into a simulator back-end
• What happens to the network if we move all our
  Exchange servers to a single data centre?
• Automatic configuration
• Close the loop enable network to meet dynamic
  SLAs
• Reconfigure network to track e.g. time of day
  load changes

6
Summary

• Anemone is a pervasive endsystem network
  management platform
• combining endsystem flow monitoring with topology
  data from standard routeing protocols
• Allows insight into network behaviour in the face
  of opaque traffic and distributed control
• Potential applications range from visualization
  to simulation and real-time control

7
Sampling?

• In an ideal world, dont sample ?
• Simply monitor all traffic everywhere
• Gives complete information about network
• but, of course, world is not ideal ?
• Operational reasons preclude 100 coverage
• NetFlow Performance, storage, etc
• Anemone Access, platform support, etc
• Do we need 100 coverage anyway?
• Existing monitoring systems sample in time
• 1/N sampling where N is e.g. 500 packets
• Source of inaccuracy in data

8
In space not time

• There will always be some traffic sources or sink
  Anemone cant monitor
• However, a trace of our enterprise network
  displayed highly asymmetric traffic patterns
• Of 15k observed source IPs, 40 saw 99.999 bytes
• A few busy endsystems (servers) observe a
  disproportionate amount of the data
• Anemone monitors both transmitted and received
  traffic
• Thus, Anemone samples in space
• Complete data on all flows at all instrumented
  endsystems
• but might not be able to instrument all
  endsystems
• but maybe we dont care?

9
In space not time

• What are the implications?
• Accuracy increases slightly faster than coverage
  (transmit receive)
• Careful selection of instrumented nodes permits
  high accuracy with low coverage (traffic
  asymmetry)
• High variance if instrumented nodes selected at
  random (traffic asymmetry)
• Get just 1 or 2 machines and you do dramatically
  better

10
Instrumentation vs. coverage
11
For the sampling experts ?

• How should we trade off overheads vs. accuracy?
• Auto-select nodes to monitor based on query
• Gracefully degrade accuracy by dropping nodes
  that are too busy to take part right now
• Perhaps end up also sampling in time?
• Still have complete information on measured
  flows, but no longer measure all flows at an
  instrumented machine
• More generally, how best to use this data?
• to infer better traffic matrices?
• to correct for uninstrumented endsystems?
• to build traffic models for e.g. anomaly
  detection?
• and not just the traffic data
• Anemone has complete IP topology
• Each flow record has endsystem specific
  information
• E.g. Application name, PID, user, execution
  context,

12
(No Transcript)