CampusWide Network Performance Monitoring and Recovery CPR

1
Campus-Wide Network Performance Monitoring and
Recovery (CPR)

• Warren Matthews (OIT/ART/RNOC)

2
Overview

• Brief overview and details of deployment
• Welcome your input
• Testbed for tools
• Other Collaboration
• Development of Analysis

3
Network Support

• Campus
• Backbone group maintain 180 buildings, 1700
  switches, 55000 ports.
• Southern Crossroads gigapop (SOX)
• Provides connectivity for 20 Universities
  throughout the South East
• 10Gbps link to Abilene backbone.

4
Motivation

• Measurement Infrastructure typically means WAN
  monitoring
• But problems are LAN and host based
• Network Operations
• Single point of view
• Catastrophic failure is easier to detect
• Little quantitative data to troubleshoot
  performance problems

5
CPR

• Campus-wide Network Performance Monitoring and
  Recovery
• Regular tests across campus network
• Active monitoring
• Donated hardware
• Cheap
• Flakey
• Scavenging of parts before donation

6
CPR

• RHEL
• State-wide license
• Control of Network
• Firewall
• Physical access
• Control of satellite server, DNS

7
Deployment

• 50 hosts on Campus
• Collocated with switches in data closets
• Multiple views of the network
• Especially the users view

8
Deployment
Boggs
ET
EDI
GLC
SOX
Savannah
GTL
Yamacraw
Class
505
TechSq Classroom
Weber
Servernet
Gateway Routers
Savant
Rich133
Savant44
Rich
Rich2
Cherry-Emerson
Daniel
GTRI
MARC
Arch
Admin
EST
Core Routers
811
Couch
845
MiRC
Skiles
SEB
SSC
Neely
NI
SI
MRDC
DMSmith
IBB
Habersham
Ajax
OHR
Sc-class
Lyman
Mason
French-class
FAB
OKeefe
King
GCATT
LAWN
Howey
French
OHR
Lib-class
Lyman
9
Toolset

• No in-house development of measurement tools.
• Original plan also didnt include much
  visualization.
• Inconvenient to click through numerous graphs

10
Measurements

• Currently
• Smokeping - roundtrip time and graphs.
• Nagios - Services.
• Security - nessus and nmap.
• Also available
• Iperf (bwctl) - TCP throughput only.
• Pathchar, traceroute

11
Measurements

• Wishlist
• OWAMP - NTP
• GOAT/Netflow
• Syslog
• Integrate with SPAM/SWARM
• Coming soon
• NDT/NPAD (central, distributed)
• Test bed for tools under development

12
Analysis

• Analysis
• Create base-lines for historical comparison
• Use multiple view to detect location
• Middleware
• Alarm system
• Plateau detector (AMP), RIPE-TT
• How should we react to alarms?
• Troubleshooting guide

13
Visualization

• Smokeping, nagios
• Built in graphs, tables
• Phplot
• Powerful graphing tool
• myCPR
• Configurable alarms and graphs

14
Case Studies

• CPR has helped solve numerous issues
• Firewall
• Network slowness for file sharing
• Dropped sessions
• Not everything is a network issue

15
GAMMON

• Georgia Measurement and Monitoring
• State-wide initiative
• Distance Learning and Professional Education
  (DLPE)
• Valdosta State University, Armstrong Atlantic
  State University, Barrow County School System.

16
GAMMON Deployment
Barrow
Bellsouth
Level3
UUNET
Qwest
SOX
GLC
GT
Peachnet
Savannah
Armstrong
Valdosta
17
Other Deployments

• Local ISPs
• Major providers (Level3, Qwest, Charter)
• Residential (SpeedFactory, BellSouth, Charter,
  Cox, Earthlink)

18
Other Deployments

• Global collaborations
• International focus in strategic plan
• Metz, Shanghai
• Leverage Global PMP Infrastructure and
  communicate using emerging standards (GGF-NMWG,
  perfSONAR)

19
Latency

• Data since January 21 2006.
• N25,790
• Mode291 ms
• Median297 ms
• Mean 318.4 ms
• Max1127 ms
• IQR27 ms

20
Routing
Transpac
SJTU
JP
CERNet
PacificWave
Abilene
GLORIAD
KRnet
SOX
GT
21
This is the end

• Contact
• Warren.matthews_at_oit.gatech.edu
• Project WebSite
• http//www.rnoc.gatech.edu/cpr