Title: SMARTxAC: A Passive Monitoring and Analysis System for HighSpeed Networks TERENA Networking Conferen
1SMARTxAC A Passive Monitoring and Analysis
System for High-Speed NetworksTERENA Networking
Conference 2006
- Pere Barlet-Ros
- Josep Solé-Pareta
- Javier Barrantes
- Eva Codina
- Jordi Domingo-Pascual
- pbarlet, pareta, jbarranp, ecodina,
jordid_at_ac.upc.edu - http//www.ccaba.upc.edu/smartxac
Acknowledgment This work has been partially
supported by CESCA (SMARTxAC agreement) and the
Spanish MEC (ref. TSI2005-07520-C03-02)
2SMARTxAC
- SMARTxAC Traffic Monitoring and Analysis System
for the Anella Científica - Operative since July 2003
- Developed under a collaboration agreement
CESCA-UPC - Tailor-made traffic monitoring system for the
Anella Científica - Main objectives
- Low-cost platform
- Continuous monitoring of high-speed links without
packet loss - Detection of network anomalies and irregular
usage - Multi-user system Network operators and
Institutions - Measurement of two full-duplex GigE links
- Connection between Anella Científica and RedIRIS
- Current load 1.5 Gbps / 270 Kpps
3Anella Científica
Measurement point 2 x GigE full-duplex
4Daily Network Usage
5System Architecture
- Monitoring high-speed links is challenging
- Collection of Gbps and storage of Terabytes of
data per day - Limitations of current technology
- CPU power, memory access speeds, bus and disk
bandwidth, storage capacity, etc. - Tailor-made system divided according to real-time
constraints and running on different computers - Capture System (severe real-time constraints)
- Traffic Analysis System (soft real-time
constraints) - Result Visualization System (user driven)
- Data reduction Early discard unnecessary
information - Improve performance
- Reduce storage requirements
6Measurement Scenario
ANELLA CIENTÍFICA
GÉANT
Global Internet
Juniper M-20 (RedIRIS)
ESPANIX
2 x 2Gbps
REDIRIS Other Regional Nodes
RedIRIS
RedIRIS (Madrid)
CISCO 6513 (Anella Científica)
Private network
2 Gbps
Management network
dag0
2 Gbps
Internet Connection
dag1
Capture System(DAG 4.3GE GPS)
Traffic Analysis System (Linux)
Result Visualization System
7Capture System
- Capture hardware
- Intel Xeon 2.4 GHz. 1 GB. RAM
- 2 x Endace DAG 4.3GE
- 4 x Optical splitters
- Precise timestamping using GPS (Trimble Acutime
2000) - Capture software
- Multi-threaded implementation
- Collection of packet-headers without loss (no
sampling) - 5-tuple flow aggregation
- Aggregated flows are sent to the Analysis System
- Data Reduction
- Header collection 110 (90 GB/min? 9 GB/min)
- Flow aggregation 1200 (45 GB/5 min? 200 MB/5
min) - Some data is kept to analyze anomalies (window of
20 GB.)
8Measurement Scenario
ANELLA CIENTÍFICA
GÉANT
Global Internet
Juniper M-20 (RedIRIS)
ESPANIX
2 x 2Gbps
REDIRIS Other Regional Nodes
RedIRIS
RedIRIS (Madrid)
CISCO 6513 (Anella Científica)
Private network
2 Gbps
Management network
dag0
2 Gbps
Internet Connection
dag1
Capture System(DAG 4.3GE GPS)
Traffic Analysis System
Result Visualization System
9Traffic Analysis System
- Analysis hardware
- Pentium IV 2.6 GHz. 1 GB. RAM
- Analysis Software
- Aggregation of 5-tuple flows into classified
flows - ltsrcIP, dstIP, srcPort, dstPort, protogt ?
ltorigin, dest., appgt - Origins Institutions (also Network access
points) - Destinations External networks RedIRIS is
connected to - Bidirectional aggregation
- This classification can be useful for
charging/cost-sharing - Data reduction
- Classified flows gt11000 ( 60 GB/day ? 50
MB/day) - Compared with header traces gt 1250000 ( 13
TB/day)
10Measurement Scenario
ANELLA CIENTÍFICA
GÉANT
Global Internet
Juniper M-20 (RedIRIS)
ESPANIX
2 x 2Gbps
REDIRIS Other Regional Nodes
RedIRIS
RedIRIS (Madrid)
CISCO 6513 (Anella Científica)
Private network
2 Gbps
Management network
dag0
2 Gbps
Internet Connection
dag1
Capture System(DAG 4.3GE GPS)
Traffic Analysis System
Result Visualization System
11Result Visualization System
- Hardware
- Pentium III 450 MHz.
- Software
- Web-based graphical interface
- Institutions only have access to their own
statistics - Graphs are generated on demand
- Available graphs
- More than 300 combinations of graphs per
institution and day - Statistics are updated every 5 minutes
- Also weekly, monthly and yearly reports
12Use case 1 Port Scanning
- Traffic profile per application (bps)
13Use case 1 Port Scanning
- Traffic profile per application (flows/s)
14Use case 1 Port Scanning
- Destination port MySQL (tcp/3306)
15Use case 2 Warez Server
- Traffic profile per application (bps)
16Use case 2 Warez Server
17Use case 3 Denial-of-Service
- Traffic profile per application (bps)
18Anomaly Detection
- Threshold-based anomaly detection
- An upper and lower traffic threshold can be set
per institution - Thresholds bits/sec, packets/sec and flows/sec
- Different intervals day/night and
workday/weekend - Once an anomaly is detected additional
information is kept - Additional information can be reviewed later
offline - Profile-based anomaly detection (work in
progress) - Time-series prediction (adaptive linear filter)
- It is not needed to know the ordinary traffic
profile - Anomalies are detected when actual traffic
differs from its predicted value - Thresholds mitigate limitations of adaptive
prediction with long-term anomalies
19Identification of Network Applications
- Traffic classification in SMARTxAC is based on
port numbers - Port-based classification is no longer reliable
- P2P, dynamic ports, tunnelling, web-based
services, - We are developing a classification method based
on machine learning techniques - It learns features of traffic flows that identify
a given application - Packet payloads are only needed in the training
phase - Once the system is trained only packet headers
are needed
20Preliminary Results (Accuracy)
21Port-based vs. Machine Learning
-
- Port-based Machine learning
22Conclusions
- SMARTxAC is a tailor-made network monitoring
system that - Operates at gigabit speeds without packet loss
- It is relatively low-cost
- Provides very detailed information about the
network usage - Multi-user system network operators and
institutions - Since 2003, SMARTxAC is daily used by CESCA to
detect anomalies, attacks, performance problems,
network faults, etc. - Future work
- Anomaly detection and application identification
- Sampling, IPv6 support,
- Deployment of more measurement points in the
Anella Científica - Release the source code under an open-source
license - Collaboration with Intels CoMo
http//como.intel-research.net
23SMARTxAC A Passive Monitoring and Analysis
System for High-Speed NetworksTERENA Networking
Conference 2006
- Pere Barlet-Ros
- Josep Solé-Pareta
- Javier Barrantes
- Eva Codina
- Jordi Domingo-Pascual
- pbarlet, pareta, jbarranp, ecodina,
jordid_at_ac.upc.edu - http//www.ccaba.upc.edu/smartxac
Acknowledgment This work has been partially
supported by CESCA (SMARTxAC agreement) and the
Spanish MEC (ref. TSI2005-07520-C03-02)