On the ImPracticality of SystemCallBased IDSs

1
On the (Im)Practicality of System-Call-Based IDSs
2
Background (I)

• Ever since Stephanie Forrest first proposed
  system-call based anomaly detection, the problem
  has been abstracted as the modeling of acceptable
  system call behavior.
• At the early stage of research in system-call
  based IDS, such abstraction allows us to focus on
  the important open problem.

3
Background (II)

• However, as the modeling methods grow more and
  more mature, such abstraction draws all our
  attention to a problem that is mostly solved, and
  leaves us blind to the other practical problems
  that remains to be tackled.
• In the following, we will present some important,
  practical problems of system-call based anomaly
  detection, that have been abstracted away, and
  thus are largely ignored for a very long time.

4
How to interpose system calls?

• Almost all publications in this area have assumed
  that the facility for monitoring the system call
  interface is given.
• This assumption is of course perfectly justified
  if system call interposition is a trivial task
  well accomplished by some de facto systems.
  However, we find that the problem is not so
  trivial as it sounds.

5
System call interposition

• In the 2003 paper Traps and Pitfalls Practical
  Problems in System Call Interposition Based
  Security Tools, Garfinkel shows that system call
  interposition can be a non-trivial task to be
  done correctly.
• In addition, system call interposition can also
  mean a very significant overhead to the monitored
  process.

6
The cost of system call interposition

• As Sekar et. al. noted in the 2001 paper A Fast
  Automaton-Based Method for Detecting Anomalous
  Program Behaviors, the performance overhead of a
  system-call-based IDS can be split into the
  following two parts
• the cost of system call interposition (i.e.
  passing control to the monitor at every system
  call of the monitored process), and
• the cost of the analysis/detection performed by
  the monitor (i.e. determining whether the
  reported request for kernel service should be
  allowed every time the monitor get invoked).
• In the same paper, Sekar et. al. also mentioned
  that system call interception can incur as much
  as 100 performance overhead.
• More recent work shows that significant
  improvement in the efficiency of system call
  interposition. However, the performance overhead
  in some cases is still quite significant.

7
The cost of system call interposition (II)

• For example, consider systrace, presented in
  the 2003 paper Improving Host Security with
  System Call Policies, by Provos
• the microbenchmark result shows that the overhead
  for performing a geteuid under system call
  interposition can be much higher than 100.
• the macrobenchmark result shows that in the worst
  case, the performance overhead is 31. Though,
  note that this number includes the enforcement of
  a very simple, stateless policy.
• Such high overhead for system call interposition
  is mainly due to the context switches between the
  monitored process and the monitor responsible for
  determining whether the observed system call is
  valid.

8
System call interpositionthe hidden cost

• From the above results, we see that the cost of
  system-call interposition is not a negligible
  part of the overall performance overhead incurred
  by a system-call based IDS, especially when we
  are having models of normal process behavior that
  allows very efficient analysis at the monitor.
• Unfortunately, in many published work, the cost
  of system call interposition are treated as part
  of the base time for performance overhead. i.e.
  only the time for checking each system call is
  counted as the overhead in these papers.
• We believe this practice prevents us from having
  an accurate idea of how the proposed IDSs perform
  on real systems. It also prevents a fair
  comparison with any IDS that employs a different
  mechanism for monitoring processes.

9
System call interpositionthe hidden cost (II)

• In Formalizing sensitivity in static analysis
  for intrusion detection by Feng et. al., the
  overhead (in ) is computed with a base time
  which includes system call tracing without
  automaton operation.
• In one test case of the above paper, the overhead
  from system call tracing (as compared to the
  untraced time) is 7, while the reported
  overhead (as compared to the base time) is only
  1. For another test case, the overhead from
  system call tracing is 30, while the reported
  overhead is 28.
• In On effective model-based intrusion detection
  by Giffin et. al., the percentage overhead is
  computed with a base labeled unverified
  execution. We believe it can be interpreted as
  includes system call tracing without any
  checking at the intercepted system calls

10
System call interpositionmore hidden cost

• In the most common approach (e.g. strace, Janus),
  a separate user space monitor is needed for every
  traced process. Every time the traced process
  makes a system call, the monitor is invoked.
• However, all the interposition systems as well as
  IDSs studied measure time and space overhead when
  only one process is monitored. We believe the
  situation can be significantly worse when all
  processes in the system is monitored (and a
  super-linear increase in overhead is expected).

11
Final remark onSystem call interposition

• It is possible to significantly improve the
  performance for system call interposition by
  modifying the kernel (in fact, many existing
  mechanisms for system call interposition modifies
  the kernel one way or another). However, we
  believe modifying the kernel is difficult and
  error prone, especially if kernel source code is
  not available. This approach will also result in
  significant deployment effort.
• Finally, the need for checking all processes in
  the system, each under a different model further
  complicates the task significantly.

12
Porting to Windows?

• All published system-call based IDSs are designed
  for Linux. With Windows being one of the
  operating systems for which an effective IDS is
  most needed, it seems a very valid question to
  ask can we port all those systems to Windows?
• Unfortunately, no one has been answering that
  question. And in the following, we will show why
  the answer is mostly likely no or very
  difficult.

13
Porting to WindowsSystem call interposition
(again)

• System call interposition in Windows presents
  some new problems.
• Most existing system-call interposition tools can
  only monitor processes created by the tool, and
  does not support the attachment to existing
  processes. This makes it impossible for those
  tools (even if ported to Windows) to monitor
  Windows system processes (like svchost.exe,
  lsass.exe) created by Windows at boot up.

14
Porting to WindowsSystem call interposition
(again)

• The system-call interface in Windows is not
  documented, and is subjected to change over
  different versions.
• The former makes it difficult to design
  system-call based IDS, which requires the
  security related system calls and their arguments
  to be identified.
• The latter presents significant portability
  issues for any system call interposition scheme
  for Windows.

15
Porting to WindowsSystem call interposition
(again)

• The kernel services provided by Windows is not as
  rich as in Linux (e.g. Windows does not export a
  socket API via the system call interface). In
  Windows, a lot of functionalities are performed
  in the user space by the different system
  libraries.
• As a result, it is possible that system call
  trace in Windows will be less descriptive about
  the behavior of the monitored process (i.e. the
  call to a particular system service can be the
  result of many different actions). We believe
  this will make system-call based IDS less
  effective for Windows and more vulnerable to
  mimicry attacks.

16
Porting to WindowsDLL hell

• Almost all existing system-call based IDSs
  employs some statically computed model of the
  monitored process.
• The extensive use of DLLs in Windows simply
  mandates a mechanism for dynamically building
  this model from the model of different DLLs.
  The frequent update of DLLs also implies a
  constant need for updating the model for
  different DLLs.

17
Porting to WindowsDLL hell (II)

• Also, recent IDSs are starting to utilize call
  stack information in determining whether a system
  call should be allowed. We believe this will add
  to the problems with DLL, and exacerbate the high
  cost of performing system-call interposition
  addresses found on the stack cannot be used
  before they are mapped to a (module, offset)
  pair.
• Solutions to the above issues are technically
  feasible, but requires significant engineering
  efforts that remains to be seen.
• Modifications to the linker, loader as well as
  the format to executable files may be needed.

18
Porting to WindowsThreads

• As David Wagner has argued, threads pose yet
  another challenge, because the context-switching
  operation introduces another type of implicit
  control flow. If it were possible to reliably
  receive thread context switch events, handling
  threads would be straightforward.
• Unfortunately, building a mechanism for
  intercepting thread context switch in Windows is
  another very daunting task.