CSE 548 Advanced Computer Network SecurityIntrusion Detection

1
CSE 548 Advanced Computer Network
Security-Intrusion Detection

• Dijiang Huang
• Arizona State University

2
Agenda

• Intrusion Detection Overview
• Misuse Detection
• Anomaly Detection

3
Security Problems on Internet Connected Computers
4
(No Transcript)
5
Attack Family Interdependency
6
Existing internet Security Mechanisms

• Prevention
• Firewall
• Authentication, authorization
• IPSEC/VPN
• Access control
• Encryption
• Detection
• Auditing
• Misuse detection
• Anomaly detection
• Survivability
• Response

7
Existing internet Security Mechanisms

• Security mechanisms implement functions that help
  to prevent, detect, tolerate, respond to security
  attacks
• Prevention is ideal, but...
• Detection seeks to prevent by threat of punitive
  action
• Detection requires that the audit trail be
  protected from alteration
• If we cant completely prevent attack from
  happening, detection is the only option
• There could be attacks we cant detect, then live
  with it - survivable system
• Once detect the attack, then what? Active
  response!!!

8
Existing internet Security Mechanisms
9
Unique Aspects of Intrusion Detection Problem

• The Whole System is as Strong as Its Weakest
  Point
• The Root Cause of Intrusion Problem is Not
  Computer, But Human Being
• Ever Changing - Moving Target
• Countermeasures by adversary
• Conflicting Requirements
• Identity/authentication
• Anonymity

10
Key Concepts

• Vulnerability
• Flaws in system and/or networks that could be
  exploited to violate the security policy of
  system or network
• Examples
• strcpy() could result buffer overflow
• 3-way handshake of TCP could result
  denial-of-service
• Intrusion
• A specific execution of planed exploits of
  vulnerabilities to attempt to
• Access unauthorized information
• Manipulate unauthorized information
• Render system unreliable or unavailable
• Example
• Break-in server of payroll department
• Crash the traffic control computer system

11
Key Concepts

• Intrusion Detection (ID)
• The art and science of identify attempted
  intrusions
• Could be real-time or post-mortem
• ID usually involves
• Monitoring and analyzing both user and system
  activities
• Analyzing system configurations and
  vulnerabilities
• Assessing system and file integrity
• Ability to recognize patterns typical of attacks
• Analysis of abnormal activity patterns
• Tracking user policy violations
• Can Intrusion Detection Detect Sniffering?

12
Taxonomy of Intrusions

• Taxonomy a way to classify and refer to threats
  (and attacks) by names/categories
• Benefits avoid confusion
• Focus/coordinate development efforts of security
  mechanisms
• No standard yet
• One possibility by results/intentions first,
  then by techniques, then further by targets, etc.
• Associate severity/cost to each threat

13
Intrusion Taxonomy Example

• By results then by (high-level) techniques
• Illegal root
• Remote, e.g., buffer-overflow a daemon
• Local, e.g., buffer-overflow a root program
• Illegal user
• Single, e.g., guess password
• Multiple, e.g., via previously installed
  back-door
• Denial-of-Service
• Crashing, e.g., teardrop, ping-of-death, land
• Resource consumption, e.g., syn-flood
• Probe
• Simple, e.g., fast/regular port-scan
• Stealth, e.g., slow/random port-scan

14
Brief History of Intrusion Detection

• In The Beginning
• Manual Intrusion Detection in practice
• System administrator manually monitor users
  activity
• Ad hoc and non-scalable
• The Study of Intrusion Detection
• Was started by James P. Anderson's 1980 technical
  report
• Computer Security Threat Monitoring and
  Surveillance
• Anderson
• Introduced the notion of audit trails
• Suggested that audit trails contain vital
  information that could be valuable in tracking
  misuse and understanding user behavior
• Formed foundation of host-based intrusion and IDS
  in general

15
Brief History of Intrusion Detection

• Dr. Dorothy Denning at SRI International
• Developed Intrusion Detection Expert System
  (IDES) in early 80s
• Published An Intrusion Detection Model in 1987
• The first general intrusion detection model
• DIDS from UC Davis 1990
• DIDS (Distributed Intrusion Detection System) -
  Motivation, Architecture, and An Early Prototype
• Network Security Monitor (NSM) 1990
• UC Davis's Todd Heberlein introduced the idea of
  network intrusion detection in 1990

16
Brief History of Intrusion Detection

• GrIDS Graph-Based Intrusion Detection from UC
  Davis1996
• EMERALD Event Monitoring Enabling Responses to
  Anomalous Live Disturbances from SRI 1997
• NetSTAT from UC Santa Barbara 1998
• Bro from International Computer Science Institute
  (ICSI) 1998

17
Taxonomy of Intrusion Detection

• Based on Detection Technique
• Misuse detection
• Assumes that intrusions can be represented by a
  pattern or signature
• Low false positive rate
• Can only detect known intrusions
• Anomaly detection
• Assumes that all intrusive activities are
  necessarily anomalous
• Could potentially detect new intrusions
• High false positive rate
• Based on Source of Audit Trail
• Host based
• Network based
• Hybrid

18
Taxonomy of Intrusion Detection

• Based on Analysis Technique
• Expert systems
• Primarily used for misuse detection
• But could be used in anomaly detection as well
• Signature analysis
• Petri nets
• State transition analysis
• Statistics
• Neural networks
• Machine learning

19
Evaluation Criteria of Intrusion Detection

• Accuracy
• If an alert really reveals an intrusion?
• Can be quantitatively measured by false positive
  rate (FPR)
• Completeness
• Whether the IDS could detect all intrusions?
• Can be quantitatively measured by true positive
  rate (TPR) or false negative rate (FNR)
• Scalability
• Whether the intrusion detection can keep up with
  the growth of the network or traffic volume
• Robustness or fault tolerance
• Whether the IDS itself is resistant to attacks?
• If IDS is running on vulnerable host
• Timeliness
• How soon can the IDS detect the intrusion?
• Real-time or post-mortem?

20
Whats Next After Successful Intrusion Detection?

• You have discovered that there is an intrusion
• You might want to find out
• How it happened
• What vulnerability has been exploited
• How to fix the problem
• What about the intruders themselves?
• Will IDS tell you where the attack come from?

21
Incident Handling Lifecycle
CERT (Computer Emergency Response Team) model
22
Open Problems in Intrusion Detection

• Does There Exist an Undetectable Intrusion?
• Challenges in Wireless Networks.

23
Agenda

• Intrusion Detection Overview
• Misuse Detection
• Anomaly Detection

24
Misuse Detection

• Developed and Deployed in Early Days
• Define whats wrong through attack signatures
• Intrusion is detected through matching audit data
  to known attack signatures
• Can only detect known attacks
• Efficient
• Usually has fewer False Positives than anomaly
  detection
• The effectiveness of misuse detection is heavily
  dependent on how the attack signature is defined

25
Evidences of Detection

• Password file modified
• Four failed login attempts
• Failed connection attempts on 50 sequential ports
• User who usually logs in around 10am from ASU
  dorm logs in at 430am from a Russian IP address
• UDP packet to port 1434 (MS-SQL worm)

26
Elements of Misuse

• What Are the Elements of Misuse?
• Could be almost anything
• Run some executable
• File operation
• Could be actions done by normal user
• What Information Should Be Used for Specifying
  Misuse?
• Open/copy particular file
• i.e. cp /etc/passwd x
• Run particular executable under specified
  condition
• Its the sequence and condition that
  differentiate misuse from normal usage!

27
How To Identify Misuse Signature?

• Key Problem in Building Misuse Detection System
• Need to identify the misuse signature before
  incorporate it into misuse IDS
• No Well Developed/Accepted Method
• Common Practice
• IDS developer interviews system administrators
  and security analyst to collect
• A suite of known attack scenarios and key events
• Rule developer
• Identify the audit records that correspond to the
  scenario or key event
• Construct rules to represent the attack scenario
  based on expected audit records

28
How To Represent Misuse?

• State Transition Analysis
• Colored Petri Net
• Production Rules (Expert System)
• Sequence of system calls
• ?

29
State Transition System

• State Transition System
• An abstract machine that consists of set of
  states and transitions between states.
• Two basic types of state transition systems
• Labeled (or LTS for labeled transition system)
• Unlabelled

30
State Transition System

• State Transition System Differs from Finite State
  Automata (FSA)
• the set of states is not necessarily finite, or
  even countable
• the set of transitions is not necessarily finite,
  or even countable.

31
Unlabelled State Transition System

• Unlabelled State Transition System is
• Tuple (S, ?)
• S is a set (of states)
• ? ? S S is a binary relation over S (of
  transitions.)
• If p,q ? S
• (p,q) ? ? is usually written as p ? q.
• This represents the fact that there is a
  transition from state p to state q.

32
Labeled State Transition System

• A labeled transition system is
• tuple (S, ?, ?)
• S is a set (of states)
• ? is a set (of labels)
• ? ? S ? S is a ternary relation (of labeled
  transitions.)
• If p, q ? S and a ? ?
• (p,a,q) ? ? is written as
• This represents the fact that there is a
  transition from state p to state q with label a.
• Labels can represent different things
• Input expected
• Conditions that must be true to trigger the
  transition
• Actions performed during the transition

33
Petri Net

• Defined by Carl Adam Petri in 1962
• Extend State Machine with the Notion of
  Concurrency
• Petri Net Consists of
• Places
• May contain any number of tokens
• Transitions
• Between places, could involve more than 2 places
• More than one input, output places
• A transition is enabled if it can fire, i.e.,
  there are tokens in every input place
• When a transition fires (executes), it consumes a
  token from each of its input places, and produces
  a token on each of its output places
• Directed Arcs between places and transitions

34
Petri Net Formal Definition

• A Petri Net is
• Tuple (S,T,F,M0,W,K)
• S is a set of places
• T is a set of transitions.
• F is a set of arcs known as a flow relation which
  connects a place and a transition F ? (S T) U
  (T S)
• M0 S ? N is the initial marking
• For each place s? S, there are n? N tokens
• W F ? N is the set of arc weights
• For each arc f ?F, n? N denotes how many tokens
  are consumed from a place by a transition, or
  alternatively, how many tokens are produced by a
  transition and put into each place.
• K S ? N is the capacity restrictions
• For each place s?S, some positive number n? N
  denotes the maximum number of tokens that can
  occupy that place.

35
Petri-Net Example

• The state-transition matrix W- is T by S
  large, and represents the number of tokens
  taken by each transition from each place.
  Similarly, W represents the number of tokens
  given by each transition to each place.
• W W - W-

36
Expert Systems

• Expert System Consist of
• Knowledge base (facts)
• Could contain attack signatures
• Production rules ("if.., then..")
• Inference engine (controls how "if.., then.."
  rules are applied towards facts)
• Knowledge Base Consists of
• the representation chosen for the expert's
  knowledge declarations
• the inference engine used to process that
  knowledge.
• Inference Rule is
• Statement that has two parts, an if-clause and a
  then-clause.
• For example
• If you are smart and study hard in my class, then
  you will get A

37
Expert Systems

• Rule Based Reasoning
• Forward chaining
• Starts with the data available and uses the
  inference rules to conclude more data until a
  desired goal is reached
• Also called data driven
• Could discover new facts
• Backward chaining
• Starts with a list of goals and works backwards
  to see if there is data which will allow it to
  conclude any of these goals
• Also called goal driven
• Usually used for diagnostic

38
Misuse Representation

• It Is Desirable
• To have a way to represent all the variations of
  a misuse
• Otherwise, misuse detection would miss some
  variations
• How To Represent All the Variations of A Misuse
  Efficiently?
• Need some abstraction thats intuitive and yet
  precise
• How About List All the Variations
• In form of audit trail?

39
Case Study State Transition Analysis

• State Transition Analysis A Rule-Based Intrusion
  Detection Approach
• Published in IEEE Transaction on Software
  Engineering 1995
• Models penetrations as a series of state changes
  that lead from initial secure state to some
  compromised state
• State Transition Diagram
• Identifies
• The requirements for a penetration
• The compromise of a penetration
• Presents only the critical events that must occur
  in order to penetrate successfully

40
Case Study State Transition Analysis

• Aims to Address The Limitations of Previous Rule
  Based Misuse Detection
• Direct dependence on audit trail
• Rule audit record are one-one correspondence
• A slight variation of the same penetration could
  be missed!
• A slight variation of penetration would
  corresponds to different audit trail record
  sequence
• Inability to foresee an impending compromise
• Penetration rules are difficult to create and
  update

41
Case Study State Transition Analysis

• Assumptions
• Penetration requires minimum prerequisite access
  to the target system
• Penetration leads to some previously unheld
  ability
• Initial Secure State
• Compromised State
• Penetration
• Sequence of actions that lead system from some
  initial state to some compromised state

42
Case Study State Transition Analysis

• Signature Action
• The key action in the penetration, without which,
  the penetration wont be successful
• Represents the minimum requirements of the
  penetration
• The State Transition Diagram Consists of
  Signature Actions Only
• Dont care the non-signature actions
• A way of abstraction

43
Case Study State Transition Analysis

• Penetration scenario an example penetration
  scenario for 4.2 BSD UNIX that can be used to
  illegally acquire root privilege.

Satisfied by BSD and can be removed.

• Assertions
• The attacker must have write access to the mail
  directory
• The attacker must have execute access to cp,
  mail, touch, and chmod
• Roots mail file must not exist, or must be
  writable and
• The attacker cannot be root.

44
Case Study State Transition Analysis

• Requirements and compromised states of
  penetration scenario the violation actually
  occurs the moment that mail modifies the owner
  attribute of the mail file while the mail files
  setuid bit is enabled.

Target compromised state
Initial requirement state
Multiple steps
45
Case Study State Transition Analysis

• Intermediate state transition diagram of
  penetration scenario
• Identify the minimum number of actions that
  represent the penetration in reverse order

46
Case Study State Transition Analysis

• Intermediate state transition diagram of
  penetration scenario

47
Case Study State Transition Analysis

• Final state transition diagram of penetration
  scenario

48
Pattern Matching in Misuse Detection

• Misuse Detection Can Be Modeled As
• Pattern matching between the audit trail and
  known attack signatures
• Discrete Approximate Matching
• Allow an arbitrary number of events between
  adjacent sub-patterns
• Matching a b c d d a b a c e b c to signature a d
  a ca b c d d a b a c e b ca e e d e a e e c e e
  e

49
Misuse Example Revisited
a. cp /bin/csh /usr/spool/mail/root b. chmod
4755 /usr/spool/mail/root c. touch x d. mail
rootltx e. /usr/spool/mail/root f. Root a.
Assumes no root mail file b. Set setuid bit c.
Create empty file d. Mail root empty file e.
Execute setuid to root shell f. Get a root shell
50
Misuse Example Revisited

• Whats the Causal Relationship Between Each Step
• Who must happen before/after who? a lt b a lt
  e b lt e ???
• How to Represent it Efficiently and Precisely?
• Partial order

51
Partial Order Represented by Petri Net
52
Variation of Misuse Example

• a. touch x
• b. cp /bin/csh /usr/spool/mail/root
• c. chmod 4755 /usr/spool/mail/root
• d. mail rootltx
• e. /usr/spool/mail/root
• f. Root
• Will the Petri Net Model Detect This?

53
Misuse Detection Summary

• Advantages
• Straightforward
• Efficient in run-time (lower overhead)
• Once the attack signature is defined, the exact
  same attack will be caught
• Low false positive!!!
• Widely deployed used in real-world
• Limitations
• Can only detect known attacks
• ???
• Challenges
• How to accurately derive attack signatures?
• How to identify the signature actions?
• Depend on the deep understanding of the misuse
• Hard to automate

54
Agenda

• Intrusion Detection Overview
• Misuse Detection
• Anomaly Detection

55
Anomaly Detection

• More Interesting as Research Topic
• Define whats normal through profile
• Need a training session to build the profile
• Alert is raised if the current system behavior
  deviates too much from whats considered normal
• How much is too much?
• What if users behavior evolve?
• Has the potential to detect novel attacks
• Usually more computationally expensive
• Usually has more False Positives than misuse
  detection
• The effectiveness of anomaly detection is heavily
  dependent on how the normal behavior is modeled

56
Principles of Anomaly Detection

• Assumes
• Intrusions are necessarily abnormal in terms of
  user behavior or system behavior
• Focus on
• Finding and defining what is normal user/system
  behavior
• Build some model to represent the normal
  behavior
• Alert is raised if the current user/system
  behavior deviates too much from whats considered
  normal
• Characteristics
• Has the potential to detect novel attacks
• Usually more computationally expensive than
  misuse detection
• Usually has more False Positives than misuse
  detection
• The effectiveness of anomaly detection is heavily
  dependent on how well the normal behavior is
  modeled

57
Model Creation Techniques

• Models are created using two different methods
• Training The programs behavior is captured
  during a training period, in which, there is
  assumed to be no attacks. Another way is to craft
  synthetic inputs to simulate normal operation.
• Static analysis The information required by the
  model is extracted either from source code or
  binary code by means of static analysis.
• Training is easy, however, the model may miss
  some of the behavior and therefore produce false
  positives.
• Static analysis based models produce no false
  positives, yet dynamic libraries and source code
  availability pose problems.

58
Definitions for Model Analysis

• If a model is training based, it is possible that
  not every normal sequence is in the database.
  This results in some normal sequences being
  flagged as intrusions. This is called a false
  positive.
• If a model fails to flag an intrusion, this is
  called a false negative.
• Accuracy An accurate model has few or no false
  positives.
• Completeness A complete model has no false
  negatives.
• Convergence Rate The amount of training required
  for the model to reach a certain accuracy

59
A Visual Description of False Positives and
Completeness
Normal Behavior
Model
60
A Visual Description of False Positives and
Completeness
Normal Behavior
False Positives
Model
61
A Visual Description of False Positives and
Completeness
Normal Behavior
Model
False Negatives
62
Challenges in Anomaly Detection

• Need Accurate Representation of Whats Considered
  Normal Behavior
• If only modeled part of normal behavior
• Those (missed) normal behavior will be considered
  anomaly false positive!!!
• If the normal behavior was modeled loosely
• Some intrusions will be considered normal false
  negative!!!
• Need To Be Adaptive to Accommodate User/System
  Behavior Evolving
• User behavior does evolve
• System behavior could change due to upgrades of
  OS, library, compiler etc.

63
Anomaly Detection Modeling Techniques

• Model User Behavior
• Assumes that user behavior does have some
  invariant and distinguishing characteristics
• Could be statistical
• Could potentially detect masquerade attack
• Model System Behavior
• Some aspects of system are invariant and
  distinguishing
• Could potentially be more precise and rigorous
• Wont be able to detect masquerade attack

64
Modeling Users Normal Behavior

• The SRI IDES Statistical Anomaly Detector
• Published in 1991 IEEE Symposium on Security and
  Privacy
• High Level Ideas
• The users behavior is modeled through
  multivariate statistical analysis on a number of
  measurements
• The users behavior profile is updated daily
  using most recent observed behavior measurements.
• Give recent behavior more weight
• The IDES profile can adaptively learn users
  behavior

65
IDES Statistical Anomaly Detector

• The Measurement is Represented by A Vector
  ltS1,, Sngt
• All The Measurements Are Summarized by a Scalar
• IS ltS1,, Sngt C-1 ltS1,, SngtT
• C is the correlation matrix of ltS1,, Sngt
• C-1 is the inverse of the correlation matrix of
  ltS1,, Sngt
• If considers if Si and Sj are correlated
• What if Si and Sj are completely independent?
• C-1 will be the identity matrix, and
• ISS12 Sn2

Notes The correlation matrix of n random
variables S1, ..., Sn is the n   n matrix whose
i,j entry is corr(Si, Sj).
66
IDES Statistical Anomaly Detector

• Q Measurement
• To represent some intermediate statistics about
  audit measurements from past to present
• Qn1 D(An, An1) 2-rtQn
• An represents the nth appropriate audit record
• D(An, An1) represents the change between An and
  An1
• r is the decay rate, which determines the
  half-life of past Q values
• t is time
• Qn is more of a measurement of near past behavior
• Historical Frequency Distribution of Q
• Divide the time to past into a number of
  intervals
• Calculate the number of occurrences of Q values
  in each interval
• Since not every measurement results to
  appropriate audit record, the frequency
  distribution of Q is not uniformly distributed

67
Anomaly Detection Based on Systems Behavior

• Rationale
• The computer system will behave differently when
  it is compromised
• Behavior of computer System is likely to be more
  stable or predictable
• Since it is programmed!

68
Immunology Inspired Anomaly Detection

• Anomaly Detection Can Be Viewed As
• Stephanie Forrest, Steven A. Hofmeyr, Anil
  Somayaji, and Thomas A. Longstaff, A Sense of
  Self for Unix Processes, IEEE SP 1996.
• Artificial immune systems
• Immune System Is Based Upon
• The ability to distinguish between self and other
• Need a Sense of Self for Computer Behavior
• Self is synonymous to normal behavior
• Other is treated as anomaly

69
Anomaly Detection Based on Notion of Self

• Self of Systems Behavior Need to
• Be relatively Stable in the sense that it
  accommodate wide range of normal operating
  conditions
• Accommodate All the Legitimate Activities
• Start/stop running programs
• Different users use the system
• Otherwise, there will be false positive in
  anomaly detection
• Be sensitive to dangerous foreign activities
• Otherwise, there will be false negative in
  anomaly detection

70
A Self Based On Sequences of System Calls

• Rationale
• All damage to system by intrusion is done through
  system calls
• Every program implicitly specifies a set of
  system call sequences
• Determined by the functions called in the program
  and their order in all possible execution path
• Short subsequence of system calls tend to be
  consistent even though the complete sequence of
  system calls of very run of the program may be
  volatile
• Short subsequence of system calls should be
  different for different programs
• The short subsequence of system calls forms a
  sense of self for the Unix process
• Model root privileged process

71
A Self Based On Sequences of System Calls

• Collect Short Sequences of System Call of length
  5,6 or 11
• Ignore any other instructions between system
  calls
• Ignore parameters passed to the system calls
• Training Stage
• Use sliding window on traces of normal behavior
  to build a database of normal patterns
• Operation Stage
• Scan new traces and looking for patterns that are
  not in the database
• If find some sequence of system call that is not
  in database
• Raise alarm

72
Anomaly Detection Based Sequences of System Calls

• Once the Database of Sequences of System Calls Is
  Ready
• Scan new traces of system calls and identify
  those mismatches
• Could Use Misuse Percentage As Threshold of
  Anomaly

73
Summary of Anomaly Detection Based On Sequence of
System Calls

• Considered a Breakthrough in Anomaly Detection
• Advantages
• Focus on more stable system behavior rather than
  volatile user behavior
• The normal system behavior could be learned
  automatically without looking at the source code
• Limitations
• Could not detect race condition exploits (Race
  condition attacks )
• The resource referenced by a name has changed
  between the time of check and time of use.
• Race attacks do not change the system calls made
  by a victim process, but only change the
  interpretation of their operands.

74
Models of Sequence of System Calls

• There Are Multiple Ways to Model the Sequence of
  System Calls
• Enumerating list all sequences that occurred
• Frequency based
• Data mining sorting through large amounts of
  data and picking out relevant information
• Hidden Markov Model (HMM)
• Different Models
• Have different power in characterizing the normal
  system behavior
• Have different computational complexity

75
Modeling Program Behavior via Static Analysis

• Previous Approaches
• Build program behavior through learning the
  running traces of the program
• Usually can not learn all the legitimate behavior
  through training
• Results in unacceptable false positive rate!
• Why Not Use The Source Code?
• All the behaviors (right or wrong) of a program
  are actually specified by the source code, right?
• If the behavior model captures all the legitimate
  program behavior
• No false positive!!!

76
Modeling Program Behavior via Static Analysis

• Program Behavior Model
• Base on sequence of system calls
• A compromise application can not do much harm
  without interacting with the operating system
• Can be automatically built via static analysis of
  the source code!
• No false positive!
• Assumptions
• No intentional array bounds overflow
• No NULL-pointer dereferencing
• No function pointer arithmetic
• No type casting between function pointers and
  other pointers
• No run-time code generation

77
Static Analysis Models

• Trivial Model
• Model legitimate system call traces as a regular
  expression S, where S is the set of all system
  calls that the program could ever use
• Contains all the possible legitimate system call
  sequences
• However, it is too loose!
• no order information between allowable system
  calls
• Contain too many impossible sequences

78
Static Analysis Models

• Callgraph Model
• Use non-deterministic finite automata (NDFA) to
  model the allowable sequences of system calls
• Built from the control flow graph of the program
• Has legitimate states and a wrong state
• A (solid line) transition in the NDFA represents
  a system call
• No false positive
• Still contain quite number of impossible
  sequences of system calls
• False negative!
• How come?

79
Callgraph Model

• A control flow graph (CFG) is extracted from the
  source code by static analysis.
• Every procedure f has an entry(f) and exit(f)
  state. At this point the graph is disconnected.
• It is assumed that there is exactly one system
  call between nodes in the automation and these
  system calls are represented by edges.
• Every function call site v, calling f, is split
  into two nodes v and v. Epsilon edges are added
  from v to entry(f) and from exit(f) to v.
• The result is a single, connected, big graph.

80
Callgraph Example
Entry point
Epsilon edges
Function call site is split into two nodes
81
Static Analysis Modeling Summary

• Key ideas
• New models for characterizing program behavior
• Static analysis lets you build models to check
  whether the program behavior is consistent with
  the source code
• Characteristics
• Has false negative
• Could miss some intrusions
• No false positive
• Never falsely accuse legitimate program behavior
  as anomaly

82
N-Gram

• Pioneering work in the field.
• Forrest et. al. A Sense of Self for Unix
  Processes, 1996.
• Tries to define a normal behavior for a process
  by using sequences of system calls.
• As the name of their paper implies, they show
  that fixed length short sequences of system calls
  are distinguishing among applications.
• For every application a model is constructed and
  at runtime the process is monitored for
  compliance with the model.
• Definition The list of system calls issued by a
  program for the duration of its execution is
  called a system call trace.

83
N-Gram Building the Model by Training

• Slide a window of length N over a given system
  call trace and extract unique sequences of system
  calls.

Example
System Call trace
Unique Sequences
Database
84
N-Gram Monitoring

• Monitoring
• A window is slid across the system call trace as
  the program issues them, and the sequence is
  searched in the database.
• If the sequence is in the database then the
  issued system call is valid.
• If not, then the system call sequence is either
  an intrusion or a normal operation that was not
  observed during training (false positive) !!

85
Experimental Results for N-Gram

• Databases for different processes with different
  window sizes are constructed
• A normal sendmail system call trace obtained from
  a user session is tested against all processes
  databases.
• The table shows that sendmails sequences are
  unique to sendmail and are considered as
  anomalous by other models.

The table shows the number of mismatched
sequences and their percentage wrt the total
number of subsequences in the user session
86
Example of n-Gram Based Modeling of Sequences of
System Calls
87
Problems with Sequence Based Approaches Cont

• Code insertion
• As long as the order in which an attacker issues
  system calls are accepted as normal, he can
  insert and run his code on the system (i.e.
  buffer overflow)

88
Limitations of n-Gram Based Modeling of Sequences
of System Calls

• We Have Shown That
• Short subsequence of system calls (called n-gram)
  can be used to form the sense of self to
  characterize the normal behavior of program or
  system
• Limitations of n-gram representation
• n has to be relatively small (i.e. 5, 6)
• The number of n-gram is exponential to n.
• Small n tends to make the model too loose
• Short n-gram fails to capture the long term
  correlation among system calls (i.e. the 1st call
  the 10th call)
• False negative in anomaly detection
• Only recognize the patterns occurred in the
  training set
• A small variation in the pattern would be
  considered anomaly
• High false positive in anomaly detection

89
Limitations of n-Gram Based Modeling of Sequences
of System Calls

• The Program
• Has well-defined Structure
• Sequence
• Branch
• Loop
• May generate arbitrarily long sequence
• May generate arbitrarily many sequences
• n-Gram Fails To Capture
• The whole program structure
• The all the possible sequence of system calls
• n-Gram Can Only Get Partial Information About
  Legitimate Sequences of System Calls!
• The root cause of all false negatives and false
  positives

90
Finite State Automata (FSA) Based Modeling

• A Finite State Automata (FSA)
• Can represent program structures (e.g. loops and
  branches) efficiently
• Can capture an infinite number of sequences of
  arbitrary length with finite storage!
• Challenges in Building FSA Based Modeling of
  System Call Sequences
• Only have incomplete sample of system call
  sequences
• How to build it automatically and efficiently?
• How to determine the states in FSA?

91
Building Finite State Automata (FSA) Based
Modeling Automatically

• Key Ideas
• The program counter (PC) is in a way an
  indication of program state
• Different PC means different program state
• PC value can be used to differentiate the
  invocation of same system calls from different
  places
• Advantages in Using PC at Run-Time
• Dont need the source code to build the model
• Be able to construct compact FSA efficiently at
  run-time

92
Example of Finite State Automata (FSA) Based
Modeling
93
Example of Finite State Automata (FSA) Based
Modeling
94
Example of Finite State Automata (FSA) Based
Modeling

• What About Sequence S0S1S3S4S5S3S4?
• S0S1S3 is not in the set of 11 3-grams
• Pure n-gram based model would report this as
  anomaly!
• But it is legitimate sequence!
• FSA is able to recognize it as legitimate!

95
Advantages of Finite State Automata (FSA) Based
Modeling

• Faster Learning (Convergence)
• FTP server example

96
Advantages of Finite State Automata (FSA) Based
Modeling

• Fewer False Positive Rate
• FTP server example

97
Improving Finite State Automata (FSA) Based
Modeling

• FSA Model Uses Program Counter As Program State
  Information
• What About Call Stack?
• Return address of system calls could be useful
  for detecting anomaly
• Buffer overflow attack would change return
  address!
• Each System Call Has A
• Virtual stack list Aa0, a1,, an-1
• where an-1 is the return address of last function
  called

98
Reading list

• Koral Ilgun, Richard A. Kemmerer, and Phillip A.
  Porras, State Transition Analysis A Rule-Based
  Intrusion Detection Approach, IEEE Transaction on
  Software Engineering 1995.
• Stephanie Forrest, Steven A. Hofmeyr, Anil
  Somayaji, and Thomas A. Longstaff, A Sense of
  Self for Unix Processes, IEEE SP 1996.
• Feng, H.H.   Kolesnikov, O.M.   Fogla, P.   Lee,
  W.   Weibo Gong, Anomaly Detection Using Call
  Stack Information in Proceedings of the 2003
  IEEE Symposium on Security and Privacy (SP03)