A Collaborative Architecture for Intrusion Detection Systems with Intelligent Agents and KnowledgeBa

1
A Collaborative Architecture for Intrusion
Detection Systems with Intelligent Agents and
Knowledge-Based Alert Evaluation

• Jinqiao Yu, Y. V. Ramana Reddy, Sentil Selliah,
  Srinivas Kankanahalli, Sumitra Reddy and
  Vijayanand Bharadwaj
• SIPLab, Concurrent Engineering Research Center
• Lane Department of Computer Science and
  Electrical Engineering
• West Virginia University
• Morgantown, WV 26506

2
Contents

• Introduction
• Network Security Threats
• Intrusion detection system
• Strengths and Weaknesses of Different IDS
  products
• IDS Collaboration Research
• Related Research
• Our Work- TRINETR
• The Collaborative Intrusion Detection Alert
  Management Architecture in TRINETR
• Collaborative Alert Aggregation
• Knowledge-based Alert Evaluation
• Alert Correlation
• Implementation and Experiments
• Future Work
• References

3

• Introduction

4

• Intrusion Attempt or Threat
• the potential possibility of a deliberate
  unauthorized attempt to
• access information,
• manipulate information, or
• render a system unreliable or unusable.

5

• A Broad View Network Security Truth and Threats

6

• Network Security Truth and Threats
• In todays Internet age, our society is growing
  increasingly dependent upon our information
  networks confidentiality, integrity, and
  availability.
• But as the complexity of todays network goes
  further, vulnerabilities inevitably exist in many
  aspects of contemporary network and software
  systems
• Inevitable vulnerabilities guarantee the
  existence of all kinds of malicious cyber
  attacks.

7

• Distributed CSCW application can be more
  vulnerable than traditional standalone
  applications.
• Security mechanism inevitably becomes an
  indispensable part of CSCW applications.

8

• The Rapidly Growing Threats

• Incidents handled by CERT

Figure 1 Number of Incidents handled by CERT
9

• Vulnerabilities Reported
• Figure 2. Number of Vulnerabilities Reported by
  CERT

10

• Intrusion Detection System (IDS)
• The hardware or software system used to monitor
  network and host activities including data flows
  and information accesses etc . and detect
  suspicious activities.

11

• Two principal approaches in IDS
• Anomaly Detection (Behavior Detection)
• Misuse Detection (Signature Detection)

12

• Anomaly Detection
• Define correct or normal, find wrong or
  suspicious
• Define and characterize correct static form
  and/or acceptable dynamic behavior of the system,
  and then to detect wrongful changes or wrongful
  behavior. (Define correct, find wrong)
• Normal Abnormal

13

• Misuse Detection
• Define wrong, find wrong
• Involves characterizing known ways to penetrate a
  system. (Define wrong, find wrong)
• Each one is usually described as a pattern.
• Monitor for explicit pattern
• This is wrong (abnormal) -- gt

14

• Strengths and Weaknesses of Different IDS products

15
Misuse Detection

• Pros
• Precise Definition of Attack Patterns
• No training time

• Cons
• Only Detect Known Attacks
• High Type II Error (False Negatives)
• High Maintenance

16
Anomaly Detection

• Pros
• Claims to detect unknown attacks
• Low Maintenance (No need to update signatures)
• Although algorithms are complex, system overhead
  is low (especially for host based products)

• Cons
• Hard to define precisely between normal and
  abnormal
• Require training time
• Can be trained to gradually accept abnormal
• High Type I (False Positive) Rate

17

• 2. Intrusion Detection Collaboration Research

18

• Current IDS Problems
• Alert Flooding
• High False Positive Rates
• Isolated Alerts
• System Integration

19

• IDS Collaboration
• An active research direction
• Main objective reduce the number of alerts by
  collaborating different IDS outputs and discard
  false alerts reduce false negatives thread
  multiple alerts etc.

20

• First IDS Collaboration Project
• IDES then refined in EMERALD
• Different approaches in alert correlation
• Qualitative Bayesian estimation technology
• Expert-system-based approach for similarity
• Consequence mechanism

21

• TRINETR Project at CERC

22

• TRINETR
• Developing an Intrusion Alert Management System
  with knowledge-based Alert Evaluation

23

• Primary Goals of the project
• Reduce Alert Overload
• Collaborate multiple IDS systems to reduce False
  Positives
• Collaborate multiple IDS systems to reduce False
  Negatives
• Correlate events to generate global and synthetic
  alert report

24

• 3. The Collaborative Intrusion Detection Alert
  Management Architecture in TRINETR

25

• TRINETR Architecture

26

• The three components of TRINETR
• Collaborative Alert Aggregation
• Knowledge-based Alert Evaluation
• Alert Correlation

27

• Part I Collaborative Alert Aggregation
• Three functions
• Alert Preprocessing
• Convert alerts into standard formats
• Alert Clustering
• Group alerts into different clusters according to
  alert source, target, time, classification
• Collaborative Alert Merging
• Merge alerts from different IDS into synthetic
  alerts
• Voting algorithm is used to solve conflicts

28

• Part II Knowledge-base Alert Evaluation
• Collaborate IDS with host and network agents
• Evaluate alerts using the matching between known
  vulnerability information and hosts , network
  information.
• Eliminate false positives and system Immune
  alerts.
• Provide Security Solutions.

29

• Knowledge-based Evaluation

30

• Two Knowledge Bases
• Vulnerability Knowledge Base
• Network and Hosts Asset Knowledge Base
• Two Types of Agents
• Host Agent
• Network Agent Collaborate with Host Agents
• Expert System Engine
• Dynamic Evaluation Process

31

• Part III Alert Correlation
• Correlate alerts from different IDS systems to
  further reduce false positives and false
  negatives.
• Generate global and synthetic alert report
• Plan to use classical statistical approach
  combined with explicit rules to correlate alert
  events
• Under implementation

32

• 4. Implementation and Experiments

33

• Implementation and Experiments
• Collaborative Alert Aggregation
• Two IDS deployed Snort and Prelude
• IDMEF Agents, Aggregation Processes are
  implemented in Perl
• Supporting Database - MySQL
• Knowledge-based Evaluation
• Jess Expert System Engine
• Evaluation processes implemented in Java
• Front-end
• PHP Web
• Tested the prototype system with around 100
  attacks

34
Management Console Snapshots
Figure 3. Alert Information
35
Figure 4. Network Information
36
Figure 5. Host Information
37
Figure 6. Network Information Open Ports
38

• 5. Future Work
• Explore other approaches to correlate alerts
• Extend CVE into a xml vulnerability database.
• Develop Attack Response Schemes.
• Knowledge-based Patch Management.
• Intrusion Tolerance.

39

• Reference
• ANDERSON, D., FRIVOLD, T. and VALDES, A. 1995
  Next-generation intrusion detection expert system
  (NIDES) a summary. SRI International Computer
  Science Laboratory Technical Report
  SRI-CSL-95-06, (May 1995).
• BUGTRAQ, Security Focus Online.
  http//www.securityfocus.com, (2003). 
• COMMON VULNERABILITIES AND EXPOSURES, The MITRE
  Corportion. http//www.cve.mitre.org, (2003).
• CUPPENS, F. 2001 Managing alerts in a
  multi-intrusion detection environment. 17th
  Annual Computer Security Applications Conference
  (ACSAC). New-Orleans, (December 2001).
• CUPPENS, F. and MIEGE, A. 2002 Alert
  correlation in a cooperative intrusion detection
  framework. In Proceedings of the 2002 IEEE
  Symposium on Security and Privacy, (2002).
• DEBAR H. and WESPI A. 2001 The
  Intrusion-Detection Console Correlation
  Mechanism. Fourth Workshop on the Recent Advances
  in Intrusion Detection (RAID 2001), (October
  2001).
• INTRUSION DETECTION MESSAGE EXCHANGE MESSAGE
  FORMAT, http//search.ietf.org/internet-drafts/dra
  ft-ietf-idwg-idmef-xml-01.txt.
• PRELUDE http//www.prelude-ids.org.
• SNORT http//www.snort.org.

40

• QUESTIONS/COMMENTS
• Please email jyu_at_csee.wvu.edu