Runtime Detection of Buffer Overflow Attack without Explicit Sensor Data Objects. Changwoo Pyo. Hong

1
Run-time Detection of Buffer Overflow Attack
without Explicit Sensor Data Objects.Changwoo
Pyo. Hongik University, Korea.
Pyo_at_hongik.ac.krTaejin Kim. University of
Illinois at Chicago. tkim_at_cs.uic.eduByungchul
Bae. National Security Research Institute, Korea.
bcbae_at_etri.re.krGyungho Lee. University of
Illinois at Chicago. glee_at_ece.uic.edu2006021184
Kim Taek Kyou
2

• Abstract.
• Introduction
• Embedding sensor
• Check stack frame inversion
• Experiment

3
Abstract.

• Presents two schemes for detecting buffer
  overflow attacks at run-time.
• Sensor embedding, which hides sensor data objects
  inside code pointers.
• Stack frame inversion checking, which detects
  attacks by inspecting processor registers.
• Implemented the schemes by extending the GCC
  toolchain.

4
Introduction.

• Any form of buffer overflow attacks.
• Eventually accesses the program counter (PC) to
  load it with the address of an attack code.
• Proceeds in two steps.
• A buffer is overflowed beyond its limit and a
  code pointer becomes compromised. The return
  address pointer in a stack frame or a function
  pointer is utilized.
• The PC is loaded from the compromised code
  pointer when the return or call instruction
  referencing the compromised code pointer is
  executed.

5
Embedding Sensors.

• Sensor embedding, which encrypts a code pointer
  after injecting a sensor object.
• In detection-oriented approaches, the sizes and
  locations of sensors determine the detecting
  power.
• If a sensor is to be hidden, the best place is
  the code pointer which the sensor monitors.
• If the return addresses are aligned on 4-byte
  boundaries, as in typical RISCs, then the two
  least significant bits are always 0.

6
Embedding Sensors.

• Key is used for the xor PC-encoding.
• Chance of a successful attack.
• 1/(2k2mW)
• W the size of code pointers.
• k sensors.
• m xor PC-encoding uses a m-bit key.
• RISC the probability of success is 1/239 ( W
  32, m 32, k 2 )

7
Checking Stack Frame Inversion.

• Stack inversion checking utilizes a run-time
  invariant property of stack frames instead of
  checking separate sensor objects.
• Stack pointer register esp always has smaller
  values than frame pointer register ebp, as long
  as programs are in normal states at run-time.
• Buffer overflow attacks frequently break the
  property.
• When a stack frame suns in the reverse direction
  from low to high memory, the stack frame is
  called to be inverted.

8
Checking Stack Frame Inversion.

• The structure of a normal stack frame.
• A call instruction saves a return address in the
  stack.
• A called function pushes the callers frame
  pointer into the stack.
• Before a return instruction is executed, the
  callers frame pointer is popped into the frame
  pointer register.
• The return address is now at the top of the
  stack.
• The return instruction transfers control to the
  instruction following the call instruction.

9
Checking Stack Frame Inversion.

• The stack frame overflowed by linear stack
  smashing.

10
Experiment.

• Enhanced the GCC 2.95 with sensor embedding and
  frame inversion checking.
• Table 1 shows the change in execution time and
  code size caused by stack frame inversion
  checking and sensor embedding.

11
Experiment.

• We have conducted an experimental buffer overflow
  attacks using the programs provided by Libsafe.
• Exploit linear buffer overflow.
• Format string vulnerability.
• Return-into-libc.
• Return-into-libc attacks non-executable stacks.
• Sensor embedding detected all of them.
• Stack frame inversion checking detected only the
  linear stack smashing attacks.