Scalable Network-based Buffer Overflow Attack Detection

1

• Scalable Network-based Buffer Overflow Attack
  Detection

Tzi-cker Chiueh Computer Science Department Stony
Brook University Stony Brook, NY,
U.S.A. chiueh_at_cs.sunysb.edu
Fu-Hau Hsu Department of Computer Science and
Information Engineering National Central
University Taoyuan, Taiwan, R.O.C. hsufh_at_csie.ncu.
edu.tw
Fanglu Guo Symantec Research Laboratory Cupertino,
CA, U.S.A. fanglu_guo_at_symantec.com
2
Virulence of Buffer Overflow Attacks

• Buffer overflow attack is arguably the most
  widely used and thus most dangerous attack method
  used today.
• Most Internet Worms use it to proliferate
  themselves.
• It accounts for more than 50 of all the security
  vulnerabilities recorded by CERT.

3
Proposed Solutions

• Compiler Transformation
• Stack Guard, RAD, Address Obfuscation
• Library Rewriting
• OS
• Non-executable Stack
• Instruction Set
• Hardware
• AMD Athlon-64

4
Discrepancy between Theory and Practice

• In theory, these efforts have largely solved the
  buffer overflow attack problem.
• In practice, however, new buffer overflow
  vulnerabilities are still discovered and reported
  on a routine basis.

5

• substantial modification
• substantial resistance
• Why?

6
A Solution to the above Dilemma --Nebula

• Nebula
• A network-based buffer overflow attack detection
  mechanism
• Observe the network traffic only to detect BOAs
• Currently version is developed for Linux
  paltforms.

7
Existing Network-based Intrusion Detection System
(NIDS)

• Misuse intrusion detection
• Zero-day BOAs
• Labor-Intensive
• Solution automatically signature-generating
  approaches
• Anomaly intrusion detection
• False Positive

8

• Generalized Signature

9
Two Factors for a Successful Buffer
Overflow-style Attack

• A successful buffer overflow-style attack should
  be able to overflow the right place (e.g. the
  place to hold a return address with the correct
  value (e.g. the address of injected code entry
  point)).

10
Non-predicable Offset and Entry Point Address
return address
buffer where the overflow start
injected code
address of injected code entry point.
offset between the beginning of the overflowed
buffer and the overflow target.
The offset and the entry point address are
non-predicable. They can not decided by just
looking the source code or local binary code.
11
Non-predicable Offset

• For performance concerns, most compilers dont
  allocate memory for local variables in the order
  they appear in the source code, sometimes some
  space may be inserted between them. (Source Code
  doesnt help)
• Different compiler/OS uses different allocation
  strategy. (Local binaries dont help)
• Address obfuscation insert random number of space
  between local variables and return address.
  (Super good luck may help)

12
Non-predicable Entry Point Address
webserver a b security
fhsu_at_ecsl
system data
0xbfffffff
environment variables
argument strings
command line arguments and environment variables
env pointers
argv pointers
argc
13
Strategies Used by Attackers to Increase Their
Success Chance

• Repeat address patterns.
• Insert NOP (0x90) operations before the entry
  point of injected code.

14
Indispensable Elements of BO-style Attacks

• The Address
• For buffer overflow attacks, it is the address of
  the entry point of injected code.

15
Linux Process Memory Layout
0xffffffff
kernel address space
0xc0000000
address space of addresses of
injected code and frame pointers
(Stack Address Zone)
user stack
esp
8M
for Shared libraries, including libc functions
0x40000000
brk
run-time heap
data and code
16
Size of Stack Address Zone

• The default maximum size of a processs user
  space stack is 8 Mbytes.
• However, according to Ditzel et al., the average
  function frame size is 28 bytes.
• Therefore, the majority of program are not
  supposed to use a 2Mbyte stack.
• In our test, a 8k stack is enough to identify all
  10 remote exploit strings.

17
Repeating Times and Values of Return Addresses
2k stack --- 0xbffffffff 0xbfffe000
18
A Property of Stack Addresses

• The leading byte of any words that contain a
  stack address corresponds to a non-printable
  ASCII character.

19
Generalized Signature

• Signature of a stack smashing buffer overflow
  attack
• If a sub-string of a traffic string could be
  interpreted as a stack address that repeats 3 or
  more times, it is alarmed as a buffer overflow
  attack string.

20

• Contextual Analysis

21
Bypassing Detection

• Patient attackers could bypass detection based on
  repeating address signature by repeating
  addresses no more than 2 times.
• PS All the 10 remote exploit code we tested
  repeat at least 4 times.
• Attackers repeat the addresses to increase their
  chance to success. In other words, it is very
  likely that without the repeat, attackers will
  fail many times before getting a successful one.

22
Unsuccessful Attacks

• Buffer overflow-style attacks will destroy
  targeted processs address space which in turn
  usually will crash the attacked process.
• In order to recycle valuable system resources, OS
  will close the sockets opened by crash processes
  automatically.
• On both Linux and Windows, when a program is
  crashed, the OS will terminate all the programs
  pending socket connections by sending out an RST
  packet to the communicating hosts on its behalf.

23
Server Termination Signature

• After forwarding a sub-string which could be
  interpreted as a single stack address, Nebula
  detects that the server closes the TCP connection
  without sending any data, then the traffic string
  is deemed as a buffer overflow attack string.
• Future traffic coming from the same hosts will be
  blocked or examined thoroughly.

24
Will Normal Traffic Behavior the Same Way?

• HTTP Protocol (RFC 2616) works in the
  request-reply way. (After the request, there will
  be a reply before the server close the
  connection)
• SMTP protocol (RFC 2821), for e-mail, and FTP
  protocol(RFC 959) use QUIT command to close a
  connection. (QUIT can not be interpreted as a
  stack address.)

25

• Payload Bypassing

26
Payload Bypassing

• Payload bypassing tries to avoid packet analysis
  for as much traffic as possible.
• Because most buffer overflow attacks take place
  during the exchange of control messages, it is
  safe to ignore the bulk of data that is
  downloaded as uninterpreted bytes.
• For example, in an FTP session, data transferred
  over the data connection can never be used to
  mount a buffer overflow attack against the FTP
  program because the FTP program does not
  interpret them.

27
Internet Traffic Statistic

• From CacheLogics measurement on USA, Europe, and
  Asia backbone in June 2004, HTTP and P2P packets
  accounted for more than 70 of the total traffic.

28
Percentage of Payload

• Percentage of payload in the traffic when each of
  the four protocols that Nebula can recognize is
  used to transfer files of a total size of 1.22
  Gbytes.

29
Number of False Positives without Payload
Bypassing

• Number of false positives under the our sample as
  reported by Nebula.
• The minimal number of times the attack pattern is
  repeated is assumed to be 1, 2, 3 or 10, and the
  stack size tested is 2Mbytes, 16Kbytes, or
  8Kbytes. In each entry the left is the number of
  false positives for RTL attacks, whereas the
  right is the number of false positives for CI
  attacks.
• The sample includes 134966 TCP connections and
  about 1.582 Gbytes of data.

30
Number of False Positives with Payload Bypassing

• The number of false positives in the test traffic
  associated with different protocols after
  applying payload bypassing is negligible even
  when the attack pattern repetition count is 1.

31
Throughput Comparison

• The throughput of Nebula under a test HTTP
  connection when different options are turned on.
  With payload bypassing, Nebula can perform buffer
  overflow attack detection and still achieve a
  throughput higher than a generic Linux router.