Automatic Program Repair With Evolutionary Computation

1
Automatic Program Repair With Evolutionary
Computation

• Westley Weimer
• Computer Science Dept.
• University of Virginia
• Charlottesville, VA 22904
• weimer_at_virginia.edu

Stephanie Forrest Dept. of Computer
Science University of New Mexico Albuquerque, NM
87131 forrest_at_cs.unm.edu
Claire Le Goues Computer Science Dept. University
of Virginia Charlottesville, VA
22904 legoues_at_virginia.edu
ThanhVu Nguyen Dept. of Computer
Science University of New Mexico Albuquerque, NM
87131 tnguyen_at_cs.unm.edu
Presented by Teodoro Rosati CIS 601, Spring
2014 March 4, 2014
The material in this paper is taken from two
original publications, titled A Genetic
Programming Approach to Automated Software
Repair (Genetic and Evolutionary Computation
Conference, 2009) and Automatically Finding
Patches Using Genetic Programming (Proceedings
of the 2009 IEEE 31st International Conference on
Software Engineering, IEEE Computer Society).
2
Whats the Problem? Finding bugs is relatively
easy

• Famous Costly Bugs
• FDIV Intel Pentium processor (1994) 500 million
• floating point unit had a flawed division table
• Y2K (1999). Cost 500 billion
• 2 digit YR storage (e.g. 95) and 00 1900
• Mars Climate Crasher (1998). Cost 125 million
• Imperial Units (lbs of force) ? Metric Units
  (Newtons)
• Many techniques and software solutions for
  detecting and mitigating software errors
• Syntactic Bug Pattern Detection
• Decompilation and Data Flow Analysis
• Automated Theorem Proving
• Model Checking

3
I Found The Bug, Heres the catchnow I have to
fix it
4
One solution

• Manual Program Repair
• Up to 90 of total cost of a software project for
  maintenance after delivery
• Modifying existing code, repairing defects,
  maintaining code during its lifecycle

• Products often are shipped with known and unknown
  bugs because lack of development resources

5
A better solution

• Automatic Program Repair

Traditional Program Analysis Methods
Evolutionary Computation (Genetic Programming)
6
Experimental Design

• Genetic Programming
• Evolves computer programs tailored to a task
• Meaning a program is modified using similar
  pathways to genetic evolution (mutation/crossover)
  
• GP techniques have been applied to unannotated
  off-the-shelf legacy C programs
• Individual variants V with the highest fitness
  are selected for continued evolution

7
Evolutionary Computation Genetic Programming

• Inspired by biological natural selection
• Endlers guppy experiment
• Diverse source population
• Guppies variously colored
• Natural selection on population
• Habitat variation, coarse vs. fine gravel
• Predator presence, sight based
• Evolution of drug resistant bacteria
• Source normal resistant bacteria
• Antibiotics promote resistant bacteria

8
Evolutionary Computation Genetic Programming

• Natural selection and programs
• Population of Program Variants
• Variants/Individuals

Insert
DELETE
SWAP
9
Genetic Programming Repair Technical Approach

• What is it doing wrong?
• Input a set of negative (-) test cases that
  characterizes a fault
• What is it supposed to do?
• Input a set of positive () test cases that
  encode functionality requirements
• Where should we change it?
• Program locations of the (-) test cases
• How should we change it?
• Insert, delete and swap program statements and
  control flow. Insertions are preferred
• When are we finished?
• First variant that passes () and (-) cases
• Minimize differences between variant and original

10
Automatic Program Repair Representation
AST

• Abstract Syntax Tree (AST)
• C programs
• Genes Statements are basic units
• Conditional if (xgty) max x
• Expressions within a statement
• max x
• Selection Actions
• Insert, Delete, Swap of Genes
• ____________________________
• If ( x gt y )
• max x

If
gt

Gene
x
x
y
max
11
Genetic Programming Mutation Operators
Insert

• Mutation Operator
• Insert, Delete or Swap a gene

DELETE
SWAP
12
Genetic Programming Crossover Operators

• Crossover Operator
• Between 2 Parent Sub-Trees
• Crossback Operator
• Between Variant V and Parent

13
Measuring Fitness

• Variants are compiled
• Testcase evaluated in virtual machine/sandbox
• Fitness measured using formula
• fitness(P) WPosT ? ? t ? PosT ? P
  passes t ?
• WNegT ? ? t ? NegT ? P passes t ?
• Note
• WPosT weight of each successful positive test
• WNegT weight of each successful negative test

14
For example

• December 31, 2008
• A bug was reported in Microsoft Zune media
  players
• Zune would freeze when the value of the input
  days is the last day of a leap year (e.g. 10,593)
  ? INFINITE LOOP!

Infinite loop
15
1. What is it doing wrong? ? Negative
Test Cases

• Negative Test Case
• input days set to 10,593 (last day of leap year)
• program executes lines 1 16
• then repeats lines 3, 4, 8 and 11 infinitely

Infinite loop
16
2. Whats it supposed to do? ? Positive
Test Cases

• Positive Test Case
• input days set to 1,000 (non-leap year)
• program executes lines 18, 11-18 once as expected

17
3. Where should we change it?

• Program Locations visited when executing the
  negative test cases
• lines 3, 4, 8 and 11

18
4. How should we change it? ? INSERT

• Insert an entire statement or gene
• stmtj is added to stmti

stmtj
19
4. How should we change it? ? DELETE

• Delete an entire statement or gene
• stmti is transformed into an empty block
  statement

stmti
stmti
20
4. How should we change it? ? SWAP

• Swap an entire statement or gene
• Second statement stmtj is chosen uniformly at
  random from anywhere in the program to replace
  stmti

stmtj
stmti
stmti
stmtj
21
5. When we are finished? ? Minimize
Differences

• Variant V program passes all the test cases
• Minimization Step to discard unnecessary changes
• Average repair time 42 seconds

22
Performance of the Zunebug Repair

• Evolution of the Zunebug repair 1 GP trial
• The darker curve plots the average fitness of the
  population
• The lighter curve plots the fitness of the
  individual V primary repair

23
Performance of the Zunebug Repair

• Evolution of the Zunebug repair with 20 positive
  and 4 negative test cases (equally weighted)
• The boxes represented the average over 70
  distinct trials
• The error bars represent one standard deviation.

24
Performance of the GP Algorithm
Eleven defects repaired by Genetic Programming
25
Performance of the GP Algorithm

• GP search time scales with weighted path size.
• 18 programs successfully repaired by GP (ave. of
  100 runs)
• x-axis log10 of the weighted path length
• y-axis log10 of the total number of fitness
  evaluations

26
Caveats

• Limitations (assumptions)
• defect is reproducible
• program behaves deterministically on test cases
• postive test cases encode program requirements
• no overlap in path taken by negative and postive
  test cases
• existing program can provide repair statements
• Evolution
• Not rigorously tested (parameter values,
  selection strategies, and operator design)
• Fault Localization
• critical to find viable fixes, but poorly
  understood
• Fitness Function
• Oversimplification
• Repair Quality
• dependent on a high-quality set of positive test
  cases

27
Future work

• Generic set of repair templates for GP as source
  code for mutations
• Extend with data structure definitions and
  variable declarations
• Assembly- and bytecode-level repairs
• Testing on more sophisticated errors
• Race conditions
• Assessing size and distribution of bugs for
  targeting

28
?
Questions