A Benchmark for Evaluating C ParserAnalysers

1
A Benchmark for Evaluating C Parser-Analysers

• Susan Elliott Sim
• University of Toronto
• simsuz_at_cs.utoronto.ca
• November 4, 2001

2
Background

• xfig 3.2.1 structured demonstration at CASCON99
• Children of xfig
• SORTIE project
• C Parser-Analyser Olympics
• Workshop at CASCON2001
• Tuesday, November 6, 245pm-530pm
• http//www.cs.utoronto.ca/simsuz/cascon2001

3
Motivation

• C is a complex language and writing
  parser-analysers for it is hard.
• Implementations of (C) parser-analysers
  typically trade-off between robustness and
  accuracy
• full parsing vs. lexical approaches
• Create a benchmark to empirically characterise
  where a parser-analyser lies in this design space

4
Motivation (cont.)
5
Test Buckets

• Accuracy (14)
• Preprocessor (3)
• C Syntax
• Data Structures
• Array
• enum
• union
• struct
• Variables
• Functions
• Templates
• Operators
• Exceptions
• Inheritance
• Namespaces

• Robustness (10)
• Incomplete Information
• Missing source, header, library
• Dialects
• GNU g
• MS Visual C
• IBM VisualAge C
• Heterogeneous Source
• C and Fortran
• Embeded SQL
• Generated Code
• lex/yacc
• Stateflow

6
Test Buckets (cont.)

• Many possible buckets not yet included
• Language features
• Analysis problems
• Quantitative tests
• Problem quantitative tests need an oracle

7
Participating Tool Teams

• Ccia, ATT (U. of Waterloo)
• Mike Godfrey and Andrew Trevors
• cppx (U. of Waterloo)
• Andrew Malton, Ian Bull, Ian Davis
• Rigi (U. of Victoria)
• Holger Kienle and Johannes Martin
• Tk-SEE/SN (U. of Ottawa)
• Sergei Marchenko and Tim Lethbridge

8
Objectives

• For tool teams to find out who has the best
  parser
• For me to find out how to make a benchmark to
  evaluate parsers

9
Preview of Results

• I found bugs in their tools, they found bugs in
  my test cases.
• Can your parser do X? I dont know, give me a
  test case.
• Im learning a lot about how to build a
  benchmark. (I dont have one yet.)
• There will be a winner
• In the land of the blind, the one-eyed parser is
  king.