Offline IP Traffic Analysis in POSTECH network Final Demo

1
Offline IP Traffic Analysis in POSTECH
network- Final Demo
??? (20042619) ??? (20042573) hyungjo, yjwon
_at_ postech.ac.kr 2004.6.12 Distributed
Processing Network Management Lab. POSTECH
2
What do we need to know beforehand?

• NG-MON (Next Generation Network Traffic
  Monitoring and Analysis System).
• Supporting Real-time network analysis (targets
  for OC-48).
• Flow based analysis system. What is a flow?
• Host throughput Analysis.
• Application Analysis.
• Security Analysis.
• We are going to reply on some of these modules.

3
Problems and what we have we done?

• So far, NG-MON and any other real-time analysis
  and monitoring systems focus on capturing and
  preliminary data analysis.
• Leave everything else to users (most likely to
  network administrators).
• We rely on our intuitions for the causes of the
  problem in case of network chaos.
• We need to verify our intuition or theory by
  providing reasonable evidence from well analyzed
  data.
• PROBLEM? Lack of in-depth traffic analysis and
  measuring metrics.
• We proposed measuring categories and described a
  process to verify each with in-depth analysis.

4
Our initial goal was

• To present POSTECH network traffic behavior
  characteristic.
• If we feel that there is a problem with our
  network, then we will bring it to focus and
  perhaps come up with possible solutions to it.
• To present the trends of current network traffic.
• We will be looking at monitored and pre-analyzed
  data from various perspectives.
• We will be looking for clues to find
  relationships between categories (metrics) which
  we suggest.
• To provide basic analysis requirements for
  traffic modeling.
• We hope to generalize the discovered properties
  and metrics for all IP-based network.

5
Architecture

• Note that we are not actually building a network
  management system (application) Ours is more
  like a process of delivering analysis results.
• Please keep that in mind.

6
Architecture Continue

• We gather data (capturing packets) at the
  internet junctions of POSTECH.
• Using optical tap. This will not affect any of
  the network performance.
• We receive raw data from the Flow Generator
  module and pre-analyzed data from the
  Application Analyzer module of NG-MON.
• A week long data consists of 10080 files
  (60247) and more than 50 GB E.g. 0201-0207,
  0217-0223, 0226-0304, 0402-0409
• 4 sets of a week long data. (Each taken from
  Feb.(2), March, April).
• Another 4 data sets from the Application Analyzer
  module Flow Duration Organizer.
• We wrote dozens of small programs to get
  statistical analysis results Statistic
  Extractor.

7
Implementation Details

• Two different flow formats are used (top vs.
  bottom)
• Reorganizing by time-stamps.
• Reducing data size.
• (50GB-gt20GB)
• Total over 8,000 lines of code and scripts.
• Result files consist of a long list of scale
  values.
• Matlab and etc. read in and draw graphs/tables.

8
Classification

• We propose various categories and verify each for
  their appropriateness as measuring metrics. Our
  final classification (with analysis) is the
  following
• Overall Analysis
• Flow Analysis
• Flow Duration
• Packets in Flow
• Bytes in Flow
• Duration vs. Bytes, Packets vs. Bytes
• Port Distribution
• New Flow Occurrence
• Single Flow vs. Pair Flow
• Application Analysis
• Overall Statistics
• Details of Selected Applications
• Traffic Variation over Time

9
Analysis Results

• We obtain graphs and statistics for each category
  and explain what they meant. Our analysis
  results are available at,
• http//congo.postech.ac.kr/course/cs607/project/d
  emo.html.
• We evaluate the relationships between
  graphs/tables and provide valuable feedback of
  POSTECH network.
• We apply the same method on different sets of
  data (different time period).
• Comparing analysis results from different data
  sets.
• In some cases, we present only interesting
  results among the data sets.

10
System and Software Requirements

• We have two file servers to store all these
  binary files (using NFS).
• We mount the file server onto our local machine
  to get the data.
• File server Redhat Linux 9.0, Pentium III, 480
  G. (X2)
• Local machines for the Flow Duration Organizer
  and the Statistic Extractor.
• Redhat Linux 9.0, Dual 2.0Ghz Pentium IV, 120 G,
  1 G memory.
• We use Matlab that is a powerful mathematical
  computation and graph tool to draw graphs.
• Also, graph tools like, OriginPro and Microsofts
  Excel.

11
Example 1. iii. c. Traffic variation over time

• Time-of-day feature.
• e-Donkey (1st in both flow and bytes) Soribada
  (2nd in flow but not high rank in bytes).
• Comparing two graphs.
• Reasons
• Difference in design.
• Popularity (number of users, e-Donkey is more
  popular).
• Next, refer to host analysis (category iv.)
• Like a relay process.
• Solution P2P cache server?

12
Example 2. iii. d. Packets vs. Bytes

• Concentrated around the lower boundary.
• means small size packets with light text
  messages.
• typical of chatting program.
• Also around the upper boundary.
• File transfer functionality.

• Around the lower boundary.
• small size control packets.
• E.g. start or stop
• Wider distribution near the upper boundary.
• Transmitting at different ratio (quality).
• Actual behavior.

13
Summary (1/2)

• More detailed characteristics.
• We now have a concrete proof of how some of the
  popular applications affect the network.
• Empirical analysis.
• No longer wild guesses.
• E.g. streaming, P2P applications.
• Highlights the importance of monitoring and
  analysis.
• A basis for security breach detection, billing,
  and etc.
• Measuring metrics and their verification.

14
What have we achieved (or not)? (2/2)

• Performance Accuracy check for existing
  systems.
• By comparing the obtained results with other
  research work.
• What have we not?
• Yet to provide an easy guideline for network
  status report.
• Sometimes, MRTG graph is enough.
• Is this enough work for network planning or
  traffic modeling?
• Future work?
• Generalization of traffic characteristics and
  their inter-relationships over all IP-based
  network.
• More categories to discover and verify.
• Re-categorization.
• Focusing on several significant application.
• Applying this in-depth analysis functionality
  into real-time traffic monitoring systems.

15
References

• 1 Stefan Saroiu, Krishna P. Gummadi, Richard J.
  Dunn, Steven D. Gribble, and Henry M. Levy, "An
  Analysis of Internet Content Delivery Systems",
  Proceedings of the Fifth Symposium on Operating
  Systems Design and Implementation (OSDI 2002),
  Boston, MA, December 2002.
• 2 Krishna P. Gummadi, Richard J. Dunn, Stefan
  Saroiu, Steven D. Gribble, Henry M. Levy,and John
  Zahorjan, Measurement, Modeling, and Analysis of
  a Peer-to-Peer File-Sharing Workload Proceedings
  of the nineteenth ACM symposium on Operating
  systems principles, October 2003.
• 3 Norbert Vicari, Stefan Kohler, and Joachim
  Charzinski, "The dependence of Internet user
  traffic characteristics on access speed",
  Proceedings of the 25th Annual IEEE Conference on
  Local Computer Networks, pp. 670 - 677, Tampa,
  Florida, Nov. 2000.
• 4 Se-Hee Han, Myung-Sup Kim, Hong-Taek Ju and
  James W. Hong, "The Architecture of NG-MON A
  Passive Network Monitoring System", LNCS 2506,
  DSOM 2002, Montreal Canada, October 2002, pp.
  16-27.
• 5 Nathaniel Leibowitz, Matei Ripeanu, and Adam
  Wierzbicki, "Deconstructing the Kazaa Network",
  3rd IEEE Workshop on Internet Applications
  (WIAPP'03), June 2003.