setuid Demystified -- Examining the API of Security Operation in OS using Formal Models

1
setuid Demystified-- Examining the API of
Security Operation in OS using Formal Models

• Hao Chen, David Wagner
• UC Berkeley
• Drew Dean
• SRI International

2
Objective

• Understand the semantics of security operation
  API in OS precisely
• Applications
• Using these system calls properly in programs
• Verifying their documentations
• Detecting inconsistency in OS kernels
• Building security properties and checking them in
  programs automatically (e.g. by modelchecker)

3
What is setuid

• Access control in Unix is based the User ID
  model
• Each process has 3 user Ids
• Real uid (ruid)
• Effective uid (euid)
• Saved uid (suid)
• Uid-setting system calls
• setuid() seteuid() setreuid() setresuid()

4
The setuid Mystery

• Uid-setting system calls are a semantic mess
• Counter-intuitive semantics
• Subtle differences among different calls
• Incompatible semantics of the same call in
  different Unix systems (e.g. Linux, Solaris,
  FreeBSD)
• Incomplete, inaccurate, or even wrong
  documentation
• Reason historical artifacts

5
Solution Formal Model

• Use a formal model to describe the user ID model
• Build an FSA where
• The states describe the user IDs of a process
• The transitions describe the semantics of the
  uid-setting system calls

6
Determine the States of the FSA

• Each state is a tuple (ruid, euid, suid)
• The range of user ID values determines the number
  of states
• Example
• A process switches between a privileged user ID
  and an unprivileged ID
• 2 user IDs 0(root), x(non-root)
• 8 states

7
Problem Difficult to Determine Transitions

• Large number of transitions. E.g.
• Range of user ID values 0, x where x!0
• Number of states 8
• Number of transitions per state
• setuid(uid) 2 transitions
• seteuid(euid) 2 transitions
• setreuid(ruid, euid) 4 transitions
• setresuid(ruid, euid, suid) 8 transitions
• Total transitions 8(2248)128
• A laborious, error-prone process

8
Determine Transitions Automatically by Simulation

• Idea Exhaustively make all system calls at each
  state

For each state s(ruid, euid, suid) where
ruid, euid, suid ? 0, uid1 , uid2 , For
each system call c? setuid(e),
seteuid(e), setreuid(r,e), setresuid(r,e,s)
Make the system call c in the state s
Observe the ensuing state s Add
the transition
9
FSA for setuid() in FreeBSD
10
FSA for setuid() in Linux
11
FSA for setreuid() in Linux
12
FSA for setresuid() in Linux
13
Benefits

• Correctness the FSA reflects what programs
  experience
• Efficiency the automatic method is portable to
• Different Unix systems
• Different kernel versions

14
Application Understanding the semantics of the
system calls

• Find subtle semantic differences
• Among different uid-setting system calls
• Among the same system call on different Unix
  systems
• Find surprising, counter-intuitive semantics

15
Application Verifying Man Pages

• Incompete man page
• Man page for setuid() in Linux fails to mention
  capabilities which affect how setuid() behaves
• Wrong man pages
• FreeBSD 4.4Unprivileged users may change the
  ruid to the euid and vice versa
• Redhat Linux 7.2The setgid function checks if
  the egid of the caller and if it is the
  superuser,

16
Application Detecting Inconsistency in OS Kernel

• Linux has fsuid
• Used for filesystem permission checking
• Normally follows euid
• Invariant in Linux 2.4.18 (kernel/sys.c)
• fsuid is 0 only if at least one of ruid, euid,
  suid is 0
• Rationale
• ensuring that an fsuid-unware cross-platform
  application can automatically drop root privilege
  in fsuid by dropping it in ruid, euid, suid

17
Application Detecting Inconsistency in OS Kernel
(cont)

• A bug breaks the invariant
• The invariant is satisfied in setuid(),
  seteuid(), setreuid()
• But it is broken in setresuid()
• We found the bug using the simulator
• The bug has been confirmed by Linus and Alan and
  will be fixed using our patch.

18
Application Checking Proper Usage of Syscalls in
Programs

• Modelchecking security properties in programs
• Model a program as a PDA
• Intersect the PDA (program) with the FSA of
  uid-setting system calls to get a new PDA
• Check reachability of risky states in the new PDA
• Can answer questions like
• Can a uid-setting system call fail in this
  program?
• Can this program fail to drop privilege?
• Which part of this program run with privilege?
• Result Found known bugs in sendmail 8.10.1 and
  8.12.0

19
Conclusion

• Formal models are useful in
• Understanding the APIs of security operations
• Verifying their documentations
• Detecting inconsistency in OS kernels
• Checking proper usage of security-relevant APIs
  in programs