Defense Against The Dark Arts: Using Computer Security To Teach Core Computer Science Concepts

1
Defense Against The Dark ArtsUsing Computer
Security To Teach Core Computer Science Concepts

• Mark W. Bailey
• Associate Professor, Hamilton College
• Visiting Professor, University of Virginia

2
Outline

• Goals, Context
• Course Overview
• Tools
• Course Material
• Assignment Examples
• Core CS Concepts
• Summary

3
Observations

• Declining CS enrollments nationwide
• Need to demonstrate CS relevance
• Difficult to include new courses in limited
  curricula
• Security issues are of concern to everyone
• Future decision makers must be literate to make
  informed policy decisions
• Computer security provides an ideal framework
  for attracting and engaging students

4
Goals

• Develop an innovative set of computer security
  courses
• Include courses for students without computer
  programming expertise
• Courses should be suitable across the spectrum of
  colleges and universities
• Package and document courses to facilitate
  widespread adoption

5
Participating Schools

• Hamilton College (Liberal Arts)
• Mark Bailey
• University of Virginia(Research University)
• Jack Davidson
• Virginia State University(HBCU)
• Jeff Zadeh

6
Proposed Courses
7
Proposed Courses

• C1 and C4 require no programming experience
• Suitable for liberal arts and engineering
  students wishing to obtain technical literacy
• Use case study approach

• CS majors could take all four to obtain a strong
  background in security
• Courses C2 and C3 have programming assignments

8
Outline

• Goals, Context
• Course Overview
• Tools
• Course Material
• Assignment Examples
• Core CS Concepts
• Summary

9
C2 Defense Against The Dark Arts

• Focuses on anti-virus principles and techniques
• Prerequisites Programming and assembly
• Reinforces assembly language
• Introduces programming vulnerabilities
• Emphasizes virus prevention, detection, and
  disinfection
• Designed for third and fourth year students

10
Anti-Virus Course Motivation

• Threats to computers systems, such as viruses and
  worms, are a serious problem
• CS students need to understand malware schemes
  and defenses against them
• Teaches how to detect and defeat malicious
  software
• Analyzing programs demands application of core
  theoretical concepts of CS
• Teaches these concepts in an application area of
  great topical interest

11
Outline

• Goals, Context
• Course Overview
• Tools
• Course Material
• Assignment Examples
• Core CS Concepts
• Summary

12
Tools

• Anti-virus programs need to
• Disassemble binary code
• Analyze and reason about code
• Modify, or fix code
• Reassemble binary code
• Many of these operations are performed by
  compilers

13
Phoenix Compiler Suite

• A cutting-edge suite of compilers and tools from
  Microsoft Research
• Scalable, configurable, extensible, compilation
  infrastructure
• Configurable for new tools, and purposes
• Easy insertion of plug-ins at any point in
  analysis sequence
• Well defined, APIs encouraging analysis and
  transformation reuse
• Supports binary manipulation

14
Phoenix IR Raising/Lowering

• Notice that the flow arrows go in both directions
• A binary (in EIR form) can be
• Raised all the way to HIR, transformed
• Lowered to MIR, transformed
• Lowered to LIR, transformed
• Then written back out as a new binary

15
Outline

• Goals, Context
• Course Overview
• Tools
• Course Material
• Assignment Examples
• Core CS Concepts
• Summary

16
Topics

• Introduction, ethics, threat models
• Terminology, x86 architecture
• Tools Disassembly tools, Phoenix intro
• Phoenix binary analysis tools
• Viruses Boot, interrupt, memory resident,
  executable file
• Detecting viruses, regular expressions, lex,
  Chomsky hierarchy
• Ken Thompsons Turing Award Lecture

17
Topics (Continued)

• Obfuscation, SSA form and Phoenix
• Anti-anti-virus schemes, analyzing systems
• Retroviruses, tunneling, armor, encryption,
  oligomorphic, polymorphic, metamorphic
• Software Dynamic Translation (SDT)
• Strata SDT framework SDT security applications
• Code vulnerabilities and exploits secure coding,
  static security analyzers
• Root kits

18
Antivirus Assignments

• Tricky jump illustrated (C/assembly)
• Reinforces assembly, introduces DUMPBIN
• Dumping Phoenix IR
• Demonstrates construction of Phoenix analysis and
  instrumentation tools
• Virus code detection using lex
• Finding junk instructions using SSA form
• Removing obfuscating jumps using Phoenix control
  flow information
• Using Phoenix to prevent stack smashing
• Student research presentations

19
Outline

• Goals, Context
• Course Overview
• Tools
• Course Material
• Assignment Examples
• Core CS Concepts
• Summary

20
Tricky Jump Illustrated

• Application code
• xor eax, eax
• ret
• Can be replaced with
• push offset malicious_func
• ret
• Causing a jump instead of a return
• Students build tricky jump program
• assembly language and debugging skill building

21
Virus Code Detection

• A common virus excerpt changes the IVT
• mov eax, 4CH
• mov dword ptr eax, edx
• Which register is used is irrelevant
• Disassemble executable using dumpbin
• Recognize pattern using lex
• Introduces regular expressions and their
  limitations

22
Finding Junk Instructions

• Viruses obfuscate using junk instructions
• code SSA form
• x 2 x1 2 (useless)
• y 3 y1 3
• x 4 x2 4
• y yx y2 y1x2
• Students use Phoenixs SSA to find junk

23
Removing Obfuscating Jumps

• Simple sequences like
• x 4
• y - (z x)
• z - 3
• printf(d\n, x)

• Can be obfuscated using jumps
• x 4
• goto lab2
• lab3
• z - 3
• goto lab4
• lab2
• y (z x)
• goto lab3
• lab4
• printf(d\n, x)

24
Outline

• Goals, Context
• Course Overview
• Tools
• Course Material
• Assignment Examples
• Core CS Concepts
• Summary

25
Core CS Concepts

• Viruses often detected by pattern matching
• Regular expressions in context of suspicious
  code patterns
• Code obfuscations make pattern matching
  inadequate in practice
• Chomsky language hierarchy is used to understand
  this limitation
• Equivalence of obfuscated code applies concepts
  from computability and theory of computation
• Students learn anti-virus software must often
  approximately solve an infeasible problem

26
Core CS Concepts Continued

• Pattern matching limitations suggest semantic
  analyses found in compilers
• Simple dataflow analysis and SA form for
  de-obfuscation
• Simple register allocation/assignment used to
  defeat register renumberinganother obfuscation
  technique

27
The Ongoing Battle

• Endless advances in both malicious software and
  the tools that combat it
• As each generation is defeated by security
  software, new techniques are developed that
  defeat the security tools
• Examples Armoring of viruses, obfuscation
  techniques, evolutionary viruses such as
  polymorphic, and metamorphic viruses
• This warfare between good and bad forces has
  been found to intensify student interest

28
Outcomes

• Taught twice at Virginia, once at Hamilton and
  VSU (Spring 2007)
• UVa course overenrolled (had to turn away
  students)
• Used Phoenix infrastructure as a vehicle for
  teaching anti-virus techniques (compilers in
  disguise)
• Student feedback very positive
• Students wanted more Phoenix assignments. Would
  have liked to do more with Phoenix and less with
  Lex, but I guess time was too much of a
  limitation.

29
Summary

• Course focuses on topic of concern to everyone
• Uses core CS concepts in an interesting
  application area
• Students use state-of-the-art tools to analyze
  real code (but not real viruses)
• Course theme, title, and subject helps attract
  and fill courses
• Course materials suitable at a wide range of
  institutions
• Course materials will be made available in the
  Microsoft Academic Alliance Curriculum Repository

30
For More Information

• Mark Bailey (mbailey_at_hamilton.edu)
• Jack Davidson (jwd_at_virginia.edu)
• Jeff Zadeh (jzadeh_at_vsu.edu)

31
Microsoft Research Faculty Summit 2007