Milk or Wine: Does Software Security Improve with Age?

1
Milk or WineDoes Software Security Improve with
Age?

• Andy Ozment Stuart Schechter
• Group 62 Internal Presentation
• 14 February 2006

2
Outline

• Motivation
• Methodology
• Results characterizing vulnerabilities
• Vulnerability date of introduction
• Code base evolution
• Vulnerability density
• Vulnerability half-life
• Rate of vulnerability detection
• Conclusion

3
Motivation

• Eric Rescorla. Is Finding Security Holes a Good
  Idea? Workshop on Economics and Information
  Security.May 2004 Minneapolis, MN, USA.
• Rate of vulnerability detection is not decreasing
• 4 operating systems
• All vulns found 1997-2000
• Used ICAT database (now NVD)
• ? Vuln finding effort not producing return
• Pool of undiscovered vulns is not diminishing
• Or
• Result is artifact of shortcomings inherent to
  ICAT db

4
MotivationKnow Your Enemy

• Key Question
• Is the rate of vuln detection decreasing?
• General Concern
• Software reliability field has knowledge about
  defects
• what do we know about vulnerabilities?
• Are LOC numbers correlated to vuln numbers?
• Is newer code more secure than older code?
• What is the half-life of a vuln?
• How important is old code to security?

5
Outline

• Motivation
• Methodology
• Results characterizing vulnerabilities
• Vulnerability date of introduction
• Code base evolution
• Vulnerability density
• Vulnerability half-life
• Rate of vulnerability detection
• Conclusion

6
Methodology

• Examined 7.5 years of OpenBSD vulnerabilities
• May 1998 to November 2005
• 15 versions
• 2.3 to 3.5, inclusive
• Why OpenBSD?
• Prioritizes security
• Source code available via CVS

7
Methodology

• All vulns listed on OpenBSD security page
• Additional sources ICAT/NVD, OSVDB, Bugtraq,
  X-Force
• Vulnerability birth and death dates from CVS
• Foundation version OpenBSD 2.3
• First source stable release in which vulns
  consistently documented

8
CaveatCurrent Vuln Hunting Environment

• Data set represents current vuln hunting
  environment
• Cannot encompass changes in
• Number of vuln hunters
• Effort expended by vuln hunters
• Vuln hunting fads
• Image file processing
• Microsoft
• Ideal a precise, universal measure of time
• E.g. execution time
• Our data set is a reflection of the current
  environment

9
Outline

• Motivation
• Methodology
• Results characterizing vulnerabilities
• Vulnerability date of introduction
• Code base evolution
• Vulnerability density
• Vulnerability half-life
• Rate of vulnerability detection
• Conclusion

10
ChartVersion of Introduction
Released Version Vulns
05/98 2.3 87
12/98 2.4 14
05/99 2.5 6
12/99 2.6 9
06/00 2.7 7
10 Others 17
Total 140

• OpenBSD 2.3 contributed 62 of vulns
• These vulns introduced prior to start of 7.5 year
  study

11
ChartBirth Death Versions
12
Dominance of Old Vulnerabilities

• Majority of vulns discovered were in foundation
  version
• 62 (87 of 140) of vulns discovered in 7.5
  years
• Why does foundational code dominate vuln
  discovery?
• Is it badly written?
• Does it dominate security functionality?
• Is it more interesting/easy to vuln hunters?
• Does it dominate the code base?

13
Motivation MethodologySource Code Evolution

• Does foundation version dominate the code base?
• Methodology
• Compare base versions (2.3 vs. 2.4)
• Strip comments
• Examine .c and .h files
• Use diff tool
• Ignore whitespace and end-of-lines

14
ChartSource Code Evolution
15
Source Code Evolution

• After 7.5 years and 15 releases, the foundation
  version
• Contributed 62 of vulnerabilities
• Constituted 61 of unaltered lines of code
• So...
• Are lines of code correllated with number of
  vulns?

16
ScatterplotLines of Code vs Number
Vulnerabilities
No corellation.
17
Vulnerability Densities
Vers. Vulns 1M LOC Vulns1M LOC
2.3 87 10.14 8.58
2.4 14 0.42 33.04
2.5 6 0.27 21.84
2.6 9 1.05 8.55
2.7 7 0.77 9.10
2.8 0 0.40 0.00
2.9 4 2.23 1.79
3.0 5 0.62 8.00
3.1 2 0.81 2.46
3.2 2 0.33 6.03
3.3 2 0.32 6.18
3.4 0 0.83 0.00
3.5 2 1.44 1.39
3.6 0 0.74 0.00
3.7 0 0.91 0.00
Total 140 21.45 6.53

• Wide range of vuln densities
• Inflated LOC counts
• Version 2.4
• Internet Key Exchange (2)
• OpenSSL (3)
• Compare vuln densities to conventional wisdom on
  defect densities
• Vulns lt 0.033 / 1K LOC
• Defects 8-12 / 1K LOC

18
Vulnerability Half-life

• Different methods
• Lower bound assume all vulns have been found
• Constant time equal length of time after
  different releases
• Model discovery process fit a curve and project
  outwards
• Constant time 6 years after release

(days)
Version Half-life
2.3 878
2.4 1288
2.5 445
2.6 645
19
ChartVulnerability Half-life Lower Bound
20
Outline

• Motivation
• Methodology
• Results characterizing vulnerabilities
• Vulnerability date of introduction
• Code base evolution
• Vulnerability density
• Vulnerability half-life
• Rate of vulnerability detection
• Conclusion

21
Rate of Vulnerability Discovery

• Remember original motivation
• Rate of vuln detection not decreasing
• Costs of vuln hunting therefore overwhelm
  benefits
• (Rescorla 2004)
• Our data set is more precise
• Caveat reflection of current vuln hunting
  environment
• Is the rate of vuln detection in OpenBSD 2.3
  decreasing?

22
ChartRaw Data
Mean 29.1 Median 18 ? 29.1 Range 1 - 126
Days Since Study Began
23
ChartEight Slices of Raw Data
Days Since Study Began
24
ChartNumber of Vulns per Study Eighth

• Confidence intervals derived from a normal
  approximation of a homogenous Poisson process

25
ChartTwo Slices of Raw Data
Days Since Study Began
26
ChartNumber of Vulns per Study Half
Confidence intervals derived from a normal
approximation of a homogenous Poisson process
27
ChartFIve Cuts of Raw Data
Days Since Study Began
28
ChartTime-Between-Failures
29
Reliability Growth Modeling

• Rescorla argued vuln discovery rate was not
  decreasing
• Used Laplace test
• Attempted to fit reliability growth models
• Laplace test
• Assumes vuln discovery is non-homogenous Poisson
  process
• H0 No trend
• H1 Trends towards increasing rate of vuln
  discovery
• H2 Trends towards decreasing rate of vuln
  discovery

30
ChartLaplace Test for Trend
31
Reliability Growth Models

• Tested 8 models
• Musas Logarithmic model fit
• Acceptable one-step-ahead predictive accuracy
• Intensity
• Start 0.051 vulns/day
• End 0.024 vulns/day
• Proportion of total vulns that have been
  discovered 67.6
• Current MTTF 57.7 days

32
Conclusion

• Foundation version dominates
• 62 of vulnerabilities
• Half-life lower bound of 2.6 years
• 61 of code base at the end of study
• Rate of detection is decreasing
• Visual assessment of assumed homogenous Poisson
• Laplace test
• Musas Log. model estimates 67.6 of vulns have
  been found

33
Questions

• Questions?

34
Bibliography

• Andy Ozment and Stuart Schechter. Milk or Wine
  Does Software Security Improve with Age? Under
  consideration.
• Eric Rescorla. Is Finding Security Holes a Good
  Idea? In Workshop on Economics and Information
  Security. May 2004 Minneapolis, MN, USA.

35
Motivation Better to be found by the good guys

• Better to be found by good guys than by bad guys
• Implication
• There exists a pool of vulns in every product
• As that pool is diminished, finding vulns becomes
  more difficult
• Bad guys and good guys are both less likely to
  find vulns
• Rate of detection decreases (all else equal)

Vulns repaired, diminishing remaining pool
Finding vulns becomes more difficult
Rate of vuln detection decreases

• Eric Rescorla. Is Finding Security Holes a Good
  Idea? WEIS 2004

36
Motivation

• Does the rate of vulnerability detection in
  softwaredecrease as the software ages?
• Questions
• Does the rate of discovery diminish
  significantlyduring the products lifespan?
• Finding vulns imposes costs on users

37
MotivationBetter to be found by the good guys

• Better to be found by good guys than by bad guys
• There exists a pool of vulns in every product
• As pool is diminished, finding vulns becomes more
  difficult
• Rate of detection decreases (all else equal)
• But
• Rate of detection is not decreasing (Rescorla
  2004)
• Pool is too large to be diminished during product
  life cycle
• or
• All else is not equal
• Or
• Shortcomings inherent to ICAT db used by Rescorla

38
MotivationBetter to be found by the good guys

• Better to be found by good guys than by bad guys
• There exists a pool of vulns in every product
• As pool is diminished, finding vulns becomes more
  difficult
• Rate of detection decreases (all else equal)

ifthenelsemallocfreeforwhiledoielsifre
turnnextfreeuntilmallocifthenelsemallocfr
eeforwhiledoielsifreturnnextfreeuntilm
allocifthenelsemallocfreeforwhiledoiel
sifreturnnextfreeuntilmallocifthenelsemal
locfreeforwhiledoielsifreturnnextfreeu
ntilmallocifthenelsemallocfreeforwhiledo
ielsifreturnnextfreeuntilmallocifthenel
semallocfreeforwhiledoielsifreturnnext
freeuntilmallocifuntilreturnifthenelsemal
locfreeforwhiledoielsifreturnnextfreeu
ntilmallocifthenelsemallocfreeforwhiledo
ielsifreturnnextfreeuntilmallocifthenel
semallocfreeforwhiledoielsifreturnnext
freeuntilmallocifthenelsemallocfreeforwhi
ledoielsifreturnnextfreeuntilmallocift
henelsemallocfreeforwhiledoielsifreturn
nextfreeuntilmallocifthenelsemallocfreef
orwhiledoielsifreturnnextfreeuntilmallo
cifuntilreturnifthenelsemallocfreeforwhi
ledoielsifreturnnextfreeuntilmallocift
henelsemallocfreeforwhiledoielsifreturn
nextfreeuntilmallocifthenelsemallocfreef
orwhiledoielsifreturnnextfreeuntilmallo
cifthenelsemallocfreeforwhiledoielsif
returnnextfreeuntilmallocifthenelsemalloc
freeforwhiledoielsifreturnnextfreeuntil
mallocifthenelsemallocfreeforwhiledoi
elsifreturnnextfreeuntilmallocifuntilretur
nifthenelsemallocfreeforwhiledoielsif
returnnextfreeuntilmallocifthenelsemalloc
freeforwhiledoielsifreturnnextfreeuntil
mallocifthenelsemallocfreeforwhiledoi
elsifreturnnextfreeuntilmallocifthenelsem
allocfreeforwhiledoielsifreturnnextfree
untilmallocifthenelsemallocfreeforwhiled
oielsifreturnnextfreeuntilmallocifthen
elsemallocreturn
calloc
do
else
do
39
Motivation (Rescorla 2004)

• There exists a White Hat vulnerability-hunting
  community
• Not malicious
• Not working for developer of product
• Work of the White Hat community is expensive
• Cost to vuln hunter of finding vuln
• Cost to vendor of creating and testing patch
• Patches inform bad guys, while not installed by
  all good guys
• Asserted value of White Hat efforts
• Encourages better coding
• Provides (limited) insight into software quality
• Better to be found by good guys than by bad guys