Formalizing Sensitivity in Program Models for Intrusion Detection

1
Formalizing Sensitivityin Program Modelsfor
Intrusion Detection

• Henry Hanping Feng
• Yong Huang
• University of Massachusetts
• hfeng,yhuang_at_ecs.umass.edu

Jonathon T. Giffin Somesh Jha Barton P.
Miller University of Wisconsin giffin,jha,bart
_at_cs.wisc.edu
Wenke Lee Georgia Instituteof
Technology wenke_at_cc.gatech.edu
2
Important Ideas

• Formalizing program models facilitates
  understanding comparison.
• Exposing additional program state improves
  monitoring speed model accuracy.
• VPStatic model reads programs call stack
• Dyck model instruments binary code

3
Model-Based Intrusion Detection

• Build model of correct program behavior
• Model automaton specifying all valid system call
  sequences
• Runtime monitor ensures execution does not
  violate model

User Process
Operating System
4
Model-Based Intrusion Detection

• Model must be fast to operate
• Model must accurately represent program
• Context-sensitive models restrict impossible paths

User Process
Operating System
5
Code Example

• char filename
• pid_t2 pid
• int prepare (int index)
• char buf20
• pidindex getpid()
• strcpy(buf, filename)
• return open(buf, O_RDWR)

getpid
open
6
Code Example

• void action (void)
• uid_t uid getuid()
• int handle
• if (uid ! 0)
• handle prepare(1)
• read(handle, )
• else
• handle prepare(0)
• write(handle, )
• close(handle)

getuid
prepare
prepare
write
read
close
7
NFA Model
getuid
getpid
prepare
prepare
open
write
read
close
Function action
Function prepare
8
NFA Model
getuid
getpid
open
write
read
close
Function action
Function prepare
9
Impossible Path Exploit

• void action (void)
• uid_t uid getuid()
• int handle
• if (uid ! 0)
• handle prepare(1)
• read(handle, )
• else
• handle prepare(0)
• write(handle, )
• close(handle)

getuid
prepare
prepare
write
read
close
10
PDA Model
push X
getuid
getpid
push Y
pop X
open
pop Y
write
read
close
Function action
Function prepare
11
PDA Problems

• Impossible paths still exist
• Non-determinism indicates missing execution
  information
• PDA run-time state explosion
• e-edge identifiers maintained on a stack
• Stack non-determinism is expensive
• post algorithm cubic in automaton size

push X
getuid
getpid
push Y
12
Determinize PDA

• Expand the input alphabet by exposing the stack
  operations and the target state of the transition
• fa,p,z indicates consume input a, push z on the
  stack, and transition to state p.
• ga,p,z for pop operations.
• ea,p for operations with no stack activity.
• Result in a Deterministic PDA (or DPDA).
• Exposing only stack operations we get PDA with
  deterministic stack operations (or sDPDA).

13
NFA

• State non-determinism is cheap.

State non-determinism
unlink
unlink
14
Non-Deterministic PDA

• Stack non-determinism is expensive.

State non-determinism
unlink
unlink
15
Deterministic PDA (DPDA)

• VPStatic Model
• Model exposes stack operations target states
• Possible exponential increase in model size?

State non-determinism
unlink
unlink
16
Stack-Deterministic PDA (sDPDA)

• Dyck Model
• Model exposes stack operations
• No increase in model size?

State non-determinism
unlink
unlink
17
Input Symbol Processing Complexity

• n is state count
• m is transition count
• k is PDA input alphabet size
• r is PDA stack alphabet size

18
VPStatic

• A variant of VtPath model.
• DPDA.
• Generated by static analysis of binary.
• Use Addr(state) to expose states.

19
Determinizing via Observation

• Extract return addresses from call stack into
  virtual stack list for each system call.
• Generate a bunch of input symbols for each system
  call.

20
Determinizing via Observation
char filename pid_t2 pid int prepare (int
index) char buf20 pidindex
getpid() strcpy(buf, filename) return
open(buf, O_RDWR)


Addr(C0)
Addr(C1)
Addr(C1)
getpid
open
e(open,Addr(Sopen))
e(none,Exit(prepare)) g(none,Addr(C1),Addr(C1)) e(
none,Addr(C0)) f(none,Entry(prepare),Addr(C0)) e(o
pen,Addr(Sopen))
21
Dyck Model
X
push X
getuid
getpid
Y
push Y
pop X
open
X
Y
pop Y
write
read
close
22
Dyck Model

• getuid X getpid open X read close
• getuid Y getpid open Y write close
• Matching brackets are alphabet symbols
• Expose stack operations to runtime monitor
• Language of bracket symbols is a Dyck language
• Rewrite binary to generate bracket symbols

23
Determinizing via Binary Rewriting

• Insert code to generate bracket symbols around
  function call sites
• Notify monitor of stack activity
• Determinizes stack operations

• void action (void)
• uid_t uid getuid()
• int handle
• if (uid ! 0)
• precall(X)
• handle prepare(1)
• postcall(X)
• read(handle, )
• else
• precall(Y)
• handle prepare(0)
• postcall(Y)
• write(handle, )
• close(handle)

24
Dyck Model

• Dyck model stack-determinizes PDA

Stack non-determinism
Stack deterministism
push Y
push X
Y
X
Only one valid stack configuration
push Y
push X
25
Test Programs
26
(No Transcript)
27
(No Transcript)
28
Questions?

• Henry Hanping Feng, Yong HuangUniversity of
  MassachusettsAmherst hfeng,yhuang_at_ecs.umass.ed
  u
• Jonathon T. Giffin, Somesh Jha, Barton P.
  MillerUniversity of WisconsinMadisongiffin,jha
  ,miller_at_cs.wisc.edu
• Wenke LeeGeorgia Institute of Technologywenke_at_cc
  .gatech.edu