Title: Learning Rules for Anomaly Detection of Hostile Network Traffic
1Learning Rules for Anomaly Detection of Hostile
Network Traffic
- Matthew V. Mahoney and Philip K. Chan
- Florida Institute of Technology
2Problem How to detect novel intrusions in
network traffic given only a model of normal
traffic
- Normal web server request
- GET /index.html HTTP/1.0
- Code Red II worm
- GET /default.ida?NNNNNNNNN
3What has been done
- Firewalls
- Cant block attacks on open ports (web, mail,
DNS) - Signature Detection (SNORT, BRO)
- Hand coded rules (search for default.ida?NNN)
- Cant detect new attacks
- Anomaly Detection (eBayes, ADAM, SPADE)
- Learn rules from normal traffic for low-level
protocols (IP, TCP, ICMP) - But application protocols (HTTP, mail) are too
hard to model
4Learning Rules for Anomaly Detection (LERAD)
- Associative mining (APRIORI, etc.) learns rules
with high support and confidence for one value - LERAD learns rules with high support (n) and a
small set of allowed values (r) - Any value seen at least once in training is
allowed
If port 80 and word1 GET then word3 ?
HTTP/1.0, HTTP/1.1 (r 2)
5LERAD Steps
- Generate candidate rules
- Remove redundant rules
- Remove poorly trained rules
- LERAD is fast because steps 1-2 can be done on a
small random sample (100 tuples)
6Step 1. Generate Candidate RulesSuggested by
matching attribute values
Sample Port Word1 Word2 Word3
S1 80 GET /index.html HTTP/1.0
S2 80 GET /banner.gif HTTP/1.0
S3 25 HELO pascal MAIL
- S1 and S2 suggest
- port 80
- if port 80 then word1 GET
- if word3 HTTP/1.0 and word1 GET then port
80 - S2 and S3 suggest no rules
7Step 2. Remove Redundant RulesFavor rules with
higher score n/r
Sample Port Word1 Word2 Word3
S1 80 GET /index.html HTTP/1.0
S2 80 GET /banner.gif HTTP/1.0
S3 25 HELO pascal MAIL
- Rule 1 if port 80 then word1 GET (n/r
2/1) - Rule 2 if word2 /index.html then word1
GET (n/r 1/1) - Rule 2 has lower score and covers no new values,
so it is redundant
8Step 3. Remove Poorly Trained RulesRules with
violations in a validation set will probably
generate false alarms
r (number of allowed values)
Incompletely trained rule (removed)
Fully trained rule (kept)
Train Validate Test
9Attribute Sets
- Inbound client packets (PKT)
- IP packet cut into 24 16-bit fields
- Inbound client TCP streams
- Date, time
- Source, destination IP addresses and ports
- Length, duration
- TCP flags
- First 8 application words
Anomaly score tn/r summed over violated rules,
t time since previous violation
10Experimental Evaluation
- 1999 DARPA/Lincoln Laboratory Intrusion Detection
Evaluation (IDEVAL) - Train on week 3 (no attacks)
- Test on inside sniffer weeks 4-5 (148 simulated
probes, DOS, and R2L attacks) - Top participants in 1999 detected 40-55 of
attacks at 10 false alarms per day - 2002 university departmental server traffic
(UNIV) - 623 hours over 10 weeks
- Train and test on adjacent weeks (some unlabeled
attacks in training data) - 6 known real attacks (some multiple instances)
11Experimental ResultsPercent of attacks detected
at 10 false alarms per day
12UNIV Detection/False Alarm TradeoffPercent of
attacks detected at 0 to 40 false alarms per day
13Run Time Performance(750 MHz PC Windows Me)
- Preprocess 9 GB IDEVAL traffic 7 min.
- Train test lt 2 min. (all systems)
14Anomalies are due to bugs and idiosyncrasies in
hostile codeNo obvious way to distinguish from
benign events
UNIV attack How detected
Inside port scan HEAD / HTTP\1.0 (backslash)
Code Red II worm TCP segmentation after GET
Nimda worm host www
Scalper worm host unknown
Proxy scan host www.yahoo.com
DNS version probe (not detected)
15Contributions
- LERAD differs from association mining in that the
goal is to find rules for anomaly detection a
small set of allowed values - LERAD is fast because rules are generated from a
small sample - Testing is fast (50-75 rules)
- LERAD improves intrusion detection
- Models application protocols
- Detects more attacks
16Thank you