Evaluating Static Code Analysis Tools

1
Evaluating Static Code Analysis Tools

• CSE 6324
• Ken Garlington
• November 7, 2000

2
What Kind of Tool Should be Evaluated?

• Static analysis analysis of a program carried
  out without executing the program BS 7925-1
• Analysis could be done for any number of
  purposes
• Simplifying test generation
• Enforcing consistent coding styles
• Generating program abstractions
• Creating executables
• However, this study focuses on a specific form of
  analysis
• Detection of faults that may affect program
  functionality

3
Who Are The Users?

• Mostly interested in fault detection (affecting
  reliability)
• Not the only area of interest, just the most
  critical
• Average 5 years application programming
  experience
• Average experience in target language (C) less
  than 2 years
• Also have to support legacy software in Ada
• May use other languages in future (Java, C)
• Windows NT-based development environment

4
What Tools Should Be Selected as Candidates?

• Include currently-used baseline for comparisons
• Compiler does basic code analysis to generate
  executable
• Compiler can also detect lint-class violations
• Include (at least) one pattern-matching tool
• Preference given to tools that enforce MISRA
  standard
• Does not attempt to determine specific code
  behavior
• Include (at least) one deep analysis tool
• Attempts to model code behavior (data/control
  flow)
• Also, all candidates must
• Be commercially supported
• Have an existing client base

5
What Candidates Were Selected?

• Texas Instruments (TI) CodeComposer for TMS320C67
  (baseline)
• http//dspvillage.ti.com/docs/ccstudio/ccstudiohom
  e.jhtml
• LDRA Testbed (MISRA code standards checker)
• http//www.ldra.co.uk/1999/core-info/misra.htm
• PolySpace C Verifier (abstract-interpretation
  analysis)
• http//www.polyspace.com/Product/C/index.htm

6
How Well Should The Selected Tool Perform?
7
What Is Being Used To Evaluate The Candidates?

• Web searches (also used for selection)
• Direct vendor contacts
• Benchmark Prototype C Embedded Operating
  System
• Representative of real application algorithms,
  etc.
• Built using users standard development processes
• Standard programmer profile (experience, etc.)
• Minimal testing performed to date

8
What Are The Preliminary Results?
9
What Is Left to Do?

• Get benchmark results back from PolySpace
• Preliminary results arrived 7 Nov being
  evaluated
• Get hands-on PolySpace experience (if warranted)