Defining Program Syntax

1
Defining Program Syntax
2
Syntax And Semantics

• Programming language syntax how programs look,
  their form and structure
• Syntax is defined using a formal grammar
• Programming language semantics what programs do,
  their behavior and meaning
• Semantics is harder to define

3
Outline

• Grammar and parse tree examples
• BNF and parse tree definitions
• Constructing grammars
• Phrase structure and lexical structure
• Other grammar forms

4
An English Grammar
A sentence ltSgt is a noun phrase ltNPgt, a verb ltVgt,
and a noun phrase ltNPgt. A noun phrase ltNPgt is
an article ltAgt and a noun ltNgt. A verb ltVgt
is An article ltAgt is A noun ltNgt is...
ltSgt ltNPgt ltVgt ltNPgt ltNPgt ltAgt ltNgt ltVgt
loves hateseats ltAgt a theltNgt
dog cat rat
5
How The Grammar Works

• The grammar is a set of rules that say how to
  build a treea parse tree
• ltSgt at the root of the tree
• The grammars rules define how children can be
  added at any point in the tree
• For instance, defines nodes ltNPgt, ltVgt, and
  ltNPgt, in that order, as children of ltSgt

ltSgt ltNPgt ltVgt ltNPgt
6
Parse Derivation
ltSgt ltNPgt ltVgt ltNPgt ltNPgt ltAgt ltNgt ltVgt
loves hateseats ltAgt a the ltNgt dog
cat rat
One derivation that ltSgt the dog loves the cat is
produced by the grammar rules
ltSgt ltNPgt ltVgt ltNPgt ltAgt ltNgt ltVgt
ltNPgt ltAgt ltNgt ltVgt ltAgt ltNgt ltAgt ltNgt
loves ltAgt ltNgt the ltNgt loves ltAgt ltNgt
the dog loves ltAgt ltNgt the dog loves the
ltNgt the dog loves the cat
7
Parse Tree the dog loves the cat
ltSgt
ltNPgt ltVgt ltNPgt
ltAgt ltNgt
ltAgt ltNgt
loves
dog
the
cat
the
ltSgt ltNPgt ltVgt ltNPgt ltNPgt ltAgt ltNgt ltVgt
loves hateseats ltAgt a theltNgt dog
cat rat
ltSgt ltNPgt ltVgt ltNPgt ltAgt ltNgt
loves ltAgt ltNgt the dog loves the
cat
8
Exercise 1
ltSgt
ltNPgt ltVgt ltNPgt
ltSgt ltNPgt ltVgt ltNPgt ltNPgt ltAgt ltNgt ltVgt
loves hateseats ltAgt a theltNgt dog
cat rat
ltAgt ltNgt
ltAgt ltNgt
loves
dog
the
cat
the

• Which of the following are valid ltSgt?
• the dog hates the dog
• dog loves the cat
• loves the dog the cat
• Parse
• a cat eats the rat
• the dog loves cat

9
Outline

• Grammar and parse tree examples
• BNF and parse tree definitions
• Constructing grammars
• Phrase structure and lexical structure
• Other grammar forms

10
BNF Grammar Definition

• Backus Naur Form grammar consists of four parts
• The set of tokens
• The set of non-terminal symbols
• The start symbol
• The set of productions

11
BNF Grammar Definitions Explained
ltSgt ltNPgt ltVgt ltNPgt ltNPgt ltAgt ltNgt ltVgt
loves hateseats ltAgt a theltNgt
dog cat rat
start symbol
a production
non-terminalsymbols
tokens
12
Definition, Continued

• The tokens are the smallest units of syntax
• Strings of one or more characters of program text
  
• They are atomic not treated as being composed
  from smaller parts
• The non-terminal symbols stand for larger pieces
  of syntax
• They are strings enclosed in angle brackets, as
  in ltNPgt
• They are not strings that occur literally in
  program text
• The grammar says how they can be expanded into
  strings of tokens
• The start symbol is the particular non-terminal
  that forms the root of any parse tree for the
  grammar

13
Definition, Continued

• The productions are the tree-building rules
• Each one has a left-hand side, the separator ,
  and a right-hand side
• The left-hand side is a single non-terminal
• The right-hand side is a sequence of one or more
  things, each of which can be either a token or a
  non-terminal
• A production gives one possible way of building a
  parse tree it permits the non-terminal symbol on
  the left-hand side to have the symbols on the
  right-hand side, in order, as its children in a
  parse tree

14
Alternatives (OR)

• When there is more than one production with the
  same left-hand side, an abbreviated form can be
  used
• In BNF grammar
• Gives the left-hand side (symbol),
• the separator ,
• and then a list of possible right-hand sides
  separated by the special symbol

15
Example
ltexpgt ltexpgt ltexpgt ltexpgt ltexpgt ( ltexpgt
) a b c
Note that there are six productions in this
grammar.It is equivalent to this one
ltexpgt ltexpgt ltexpgtltexpgt ltexpgt
ltexpgtltexpgt ( ltexpgt )ltexpgt altexpgt
bltexpgt c
16
Empty

• The special non-terminal ltemptygt is for places
  where you want the grammar to generate nothing
• For example, this grammar defines a typical
  if-then construct with an optional else part

ltif-stmtgt if ltexprgt then ltstmtgt
ltelse-partgtltelse-partgt else ltstmtgt ltemptygt
17
Grammar Parse Derivation

• Begin with a start symbol
• Choose a production with start symbol on
  left-hand side
• Replace start symbol with the right-hand side of
  that production
• Choose a non-terminal S in resulting string
• Choose a production P with non-terminal S on its
  left-hand side
• Replace S with the right-hand side of P
• Repeat process until no non-terminals remain.

ltSgt ltNPgt ltVgt ltNPgt ltNPgt ltAgt ltNgt ltVgt
loves hateseats ltAgt a theltNgt dog
cat rat
a cat eats the rat
ltSgt ltNPgt ltVgt ltNPgt ltAgt ltNgt ltVgt
ltNPgt ltAgt ltNgt ltVgt ltAgt ltNgt ltAgt ltNgt
eats ltAgt ltNgt a ltNgt eats ltAgt ltNgt a
cat eats ltAgt ltNgt a cat eats the ltNgt
a cat eats the rat
18
Parse Trees
ltSgt ltNPgt ltVgt ltNPgt ltNPgt ltAgt ltNgt ltVgt
loves hateseats ltAgt a theltNgt
dog cat rat

• To build a parse tree, put the start symbol at
  the root
• Add children to every non-terminal, following any
  one of the productions for that non-terminal in
  the grammar
• Done when all the leaves are tokens
• Read off leaves from left to rightthat is the
  string derived by the tree

ltSgt a cat eats the rat
ltSgt ltNPgt ltVgt
ltNPgt ltAgt ltNgt ltVgt ltAgt ltNgt ltAgt ltNgt
eats ltAgt ltNgt a ltNgt eats ltAgt ltNgt a
cat eats ltAgt ltNgt a cat eats the
ltNgt a cat eats the rat
ltSgt ltNPgt ltVgt ltNPgt ltAgt ltNgt ltVgt
ltNPgt ltAgt ltNgt ltVgt ltAgt ltNgt ltAgt ltNgt
eats ltAgt ltNgt a ltNgt eats ltAgt ltNgt a
cat eats ltAgt ltNgt a cat eats the ltNgt
a cat eats the rat
19
A Programming Language Grammar
ltexpgt ltexpgt ltexpgt ltexpgt ltexpgt ( ltexpgt
) a b c

• An expression can be
• the sum of two expressions,
• or the product of two expressions,
• or a parenthesized subexpression,
• or a,
• or b,
• or c

20
Parse and Parse Tree abc
ltexpgt ltexpgt ltexpgt a ltexpgt a
ltexpgt ltexpgt a b c
ltexpgt
ltexpgt ltexpgt
ltexpgt ltexpgt
a
b
c
ltexpgt ltexpgt ltexpgt ltexpgt ltexpgt ( ltexpgt
) a b c
21
Parse and Parse Tree ((ab)c)
ltexpgt ( ltexpgt ) ( ltexpgt ltexpgt ) ((
ltexpgt) ltexpgt ) (( ltexpgt ) c ) ((
ltexpgt ltexpgt ) c ) (( a b ) c )
ltexpgt
( ltexpgt )
ltexpgt ltexpgt
( ltexpgt )
c
ltexpgt ltexpgt
a
b
ltexpgt ltexpgt ltexpgt ltexpgt ltexpgt ( ltexpgt
) a b c
22
Exercise 2
ltexpgt ltexpgt ltexpgt ltexpgt ltexpgt ( ltexpgt
) a b c

• Parse each of these strings
• ab
• abc
• (ab)c
• Give the parse tree for each of these strings
• ab
• abc
• (ab)c

23
Compiler Note

• What we just did is parsing trying to find a
  parse tree for a given string
• Thats what compilers do for every program you
  try to compile try to build a parse tree for
  your program, using the grammar for whatever
  language you used
• Take a course in compiler construction to learn
  about algorithms for doing this efficiently

24
Language Definition

• We use grammars to define the syntax of
  programming languages
• The language defined by a grammar is the set of
  all strings that can be derived by some parse
  tree for the grammar
• As in the previous example, that set is often
  infinite (though grammars are finite)
• Constructing grammars is a little like
  programming...

25
Outline

• Grammar and parse tree examples
• BNF and parse tree definitions
• Constructing grammars
• Phrase structure and lexical structure
• Other grammar forms

26
Constructing Grammars

• Most important trick divide and conquer
• Example the language of Java declarations
• a type name,
• a list of variables separated by commas,
• and a semicolon
• Each variable can optionally be followed by an
  initializer

float aboolean a,b,cint a1, b, c12
27
Example, Continued
int a1, b, c12

• Easy if we postpone defining the comma-separated
  list of variables with initializers
• Primitive type names are easy enough too
• (Note skipping constructed types class names,
  interface names, and array types)

ltvar-decgt lttype-namegt ltdeclarator-listgt
lttype-namegt boolean byte short int
long char float double
28
Example, Continued

• That leaves the comma-separated list of variables
  with initializers
• Again, postpone defining variables with
  initializers, and just do the comma-separated
  list part

int a1, b, c12
ltvar-decgt lttype-namegt ltdeclarator-listgt
ltdeclarator-listgt ltdeclaratorgt
ltdeclaratorgt , ltdeclarator-listgt
29
Example, Continued
int a1, b, c12

• That leaves the variables with initializers
• For full Java, we would need to allow pairs of
  square brackets after the variable name
• There is also a syntax for array initializers
• And definitions for ltvariable-namegt and ltexprgt

ltvar-decgt lttype-namegt ltdeclarator-listgt
ltdeclarator-listgt ltdeclaratorgt
ltdeclaratorgt , ltdeclarator-listgt
ltdeclaratorgt ltvariable-namegt
ltvariable-namegt ltexprgt
30
Grammar Construction Example

• Construct a grammar in BNF for each language
• ltdigitgt as a character 0-9.
• ltunsignedgt as the set of all strings with one or
  more ltdigitgt. Note the left-recursion.
• ltsignedgt as the set of all strings starting with
  or and followed by an ltunsignedgt.

ltdigitgt 0 1 2 3 4 5 6 7 8 9
ltunsignedgt ltdigitgt ltunsignedgt ltdigitgt
ltsignedgt ltunsignedgt -ltunsignedgt
31
Exercise 3
ltdigitgt 0 1 2 3 4 5 6 7 8 9
ltunsignedgt ltdigitgt ltunsignedgt ltdigitgt
ltsignedgt ltunsignedgt -ltunsignedgt

• Construct a grammar in BNF for each language
• ltintegergt as the set of all strings of ltsignedgt
  or ltunsignedgt.
• ltdecimalgt as the set of all strings of ltintegergt
  followed by a . and optionally followed by an
  ltunsignedgt.
• lt2or3digitsgt as the set of all strings of two or
  three ltdigitgt.
• ltAdigitBgt as the set of all strings beginning
  with A and followed by a ltdigitgt or a B.
• lt12sgt as the set of all strings beginning with
  1 and followed by any number of 2s.
• lt2s1gt as the set of all strings beginning with
  any number of 2s and followed by a 1.
• ltAdigitBsgt as the set of all strings beginning
  with A and optionally followed by any number of
  ltdigitgt or B.

32
Outline

• Grammar and parse tree examples
• BNF and parse tree definitions
• Constructing grammars
• Phrase structure and lexical structure
• Other grammar forms

33
Where Do Tokens Come From?
ltdigitgt 0 1 2 3 4 5 6 7 8 9
ltunsignedgt ltdigitgt ltunsignedgt ltdigitgt

• Tokens are pieces of program text that we choose
  not to think of as being built from smaller
  pieces
• Identifiers (count), keywords (if), operators
  (), constants (123.4), etc.
• Programs stored in files are just sequences of
  characters
• How is such a file divided into a sequence of
  tokens?

34
Lexical Structure AndPhrase Structure

• Phrase structure how a program is built from a
  sequence of tokens
• Lexical structure how tokens are built from a
  sequence of characters

ltif-stmtgt if ltexprgt then ltstmtgt
ltelse-partgtltelse-partgt else ltstmtgt ltemptygt
ltdigitgt 0 1 2 3 4 5 6 7 8 9
ltunsignedgt ltdigitgt ltunsignedgt ltdigitgt
35
One Grammar For Both

• You could do it all with one grammar by using
  characters as the only tokens
• Not done in practice things like white space and
  comments would make the grammar too messy to be
  readable

ltif-stmtgt if ltwhite-spacegt ltexprgt
ltwhite-spacegt then ltwhite-spacegt
ltstmtgt ltwhite-spacegt
ltelse-partgtltelse-partgt else ltwhite-spacegt
ltstmtgt ltemptygt
36
Separate Grammars

• Usually there are two separate grammars
• One says how to construct a sequence of tokens
  from a file of characters
• One says how to construct a parse tree from a
  sequence of tokens

ltprogram-filegt ltend-of-filegt ltelementgt
ltprogram-filegtltelementgt lttokengt
ltone-white-spacegt ltcommentgtltone-white-spacegt
ltspacegt lttabgt ltend-of-linegtlttokengt
ltidentifiergt ltoperatorgt ltconstantgt
37
Separate Compiler Passes

• The scanner reads the input file and divides it
  into tokens according to the first grammar
• The scanner discards white space and comments
• The parser constructs a parse tree (or at least
  goes through the motionsmore about this later)
  from the token stream according to the second
  grammar

38
Exercise 4
ltspacegt ltdigitgt 0 1 2 3 4 5
6 7 8 9 ltunsignedgt ltdigitgt
ltunsignedgt ltdigitgt ltsignedgt ltunsignedgt
-ltunsignedgtltintegergt ltsignedgt
ltunsignedgtltdecimalgt ltintegergt.ltunsignedgt
ltintegergt .ltoperatorgt
ltidentifiergt x yltconstantgt ltintegergt
ltdecimalgtltkeywordgt if then endif

• List the scanner output from the following
• if x 5 then y x y endif

39
Historical Note 1

• Early languages sometimes did not separate
  lexical structure from phrase structure
• Early Fortran and Algol dialects allowed spaces
  anywhere, even in the middle of a keyword
• Do 10 I 1.25 ? Do10I1.25 / Assignment
  /
• Do 10 I 1,25 ? Do10I1,25 / Loop
  /
• Other languages like PL/I allow keywords to be
  used as identifiers
• IF THEN THEN THEN ELSE ELSE ELSE THEN
• This makes them harder to scan and parse
• It also reduces readability

40
Historical Note 2

• Some languages have a fixed-format lexical
  structurecolumn positions are significant
• One statement per line (i.e. per card)
• First few columns for statement label
• Etc.
• Early dialects of Fortran, Cobol, and Basic
• Almost all modern languages are free-format
  column positions are ignored

41
Outline

• Grammar and parse tree examples
• BNF and parse tree definitions
• Constructing grammars
• Phrase structure and lexical structure
• Other grammar forms

42
Other Grammar Forms

• BNF variations
• EBNF variations
• Syntax diagrams

43
BNF Variations

• Some use ? or instead of
• Some leave out the angle brackets and use a
  distinct typeface for tokens
• Some allow single quotes around tokens, for
  example to distinguish as a token from as a
  meta-symbol

44
EBNF Variations

• Additional syntax to simplify some grammar
  chores
• x or x to mean zero or more repetitions of x
• x to mean one or more repetitions of x
• x to mean x is optional (i.e. x ltemptygt)
• ( ) for grouping
• anywhere to mean a choice among alternatives
• Quotes around tokens, if necessary, to
  distinguish from all these meta-symbols

45
EBNF Examples
ltif-stmtgt if ltexprgt then ltstmtgt else ltstmtgt
ltstmt-listgt ltstmtgt
ltthing-listgt (ltstmtgt ltdeclarationgt)
ltdigitgt 0 1 2 3 4 5 6 7 8 9
ltunsignedgt ltdigitgt
ltsignedgt (-)ltunsignedgt

• Anything that extends BNF this way is called an
  Extended BNF EBNF
• There are many variations

46
Exercise 5

• Construct a grammar in EBNF for each language
• ltunsignedgt as the set of all strings with one or
  more ltdigitgt.
• ltsignedgt as the set of all strings starting with
  or and followed by an ltunsignedgt.
• ltintegergt as the set of all strings of ltsignedgt
  or ltunsignedgt.
• ltdecimalgt as the set of all strings of ltintegergt
  followed by a . and optionally followed by an
  ltunsignedgt.
• ltidentifiergt as the set of all strings starting
  with ltalphagt and followed by zero or more ltalphagt
  or ltdigitgt.

x or x to mean zero or more repetitions of
x x to mean one or more repetitions of x x to
mean x is optional (i.e. x ltemptygt) ( ) for
grouping anywhere to mean a choice among
alternatives
EBNF Extensions
47
Exercise 5continued
x or x to mean zero or more repetitions of
x x to mean one or more repetitions of x x to
mean x is optional (i.e. x ltemptygt) ( ) for
grouping anywhere to mean a choice among
alternatives

• Construct a grammar in EBNF for each language
• lt12sgt as the set of all strings beginning with
  1 and followed by any number of 2s.
• lt2s1gt as the set of all strings beginning with
  any number of 2s and followed by a 1.
• ltAdigitBsgt as the set of all strings beginning
  with A and optionally followed by any number of
  ltdigitgt or B.
• Indiana non-vanity license plates, such as 22Z1.
• Scientific notation (e.g. 1.2E-13)

48
Syntax Diagrams

• Syntax diagrams (railroad diagrams)
• Start with an EBNF grammar
• A simple production is just a chain of boxes (for
  nonterminals) and ovals (for terminals)

ltif-stmtgt if ltexprgt then ltstmtgt else ltstmtgt
if-stmt
if
then
else
expr
stmt
stmt
49
Bypasses

• Square-bracket pieces from the EBNF get paths
  that bypass them

ltif-stmtgt if ltexprgt then ltstmtgt else ltstmtgt
if-stmt
if
then
else
expr
stmt
stmt
50
Branching

• Use branching for multiple productions

ltexpgt ltexpgt ltexpgt ltexpgt ltexpgt ( ltexpgt
) a b c
51
Loops

• Use loops for EBNF curly brackets

ltexpgt ltaddendgt ltaddendgt
52
Syntax Diagrams, Pro and Con

• Easier for people to read casually
• Harder to read precisely what will the parse
  tree look like?
• Harder to make machine readable (for automatic
  parser-generators)

53
Formal Context-Free Grammars

• In the study of formal languages and automata,
  grammars are expressed in yet another notation
• These are called context-free grammars because
  children of a node only depend on that nodes
  non-terminal symbol not on the context of
  neighboring nodes in the tree. Simpler to define
  and compile.
• Context sensitive language elements include scope
  but is not generally part of a grammar.
• Other kinds of grammars are also studied regular
  grammars (weaker), context-sensitive grammars
  (stronger), etc.

S ? aSb X S is a string of symbols a S
b or X. X ? cX ? X is a string of
symbols c X or empty
54
Many Other Variations

• BNF and EBNF ideas are widely used
• Exact notation differs, in spite of occasional
  efforts to get uniformity
• But as long as you understand the ideas,
  differences in notation are easy to pick up

55
Example
WhileStatement while ( Expression ) Statement
DoStatement do Statement while ( Expression )
ForStatement for ( ForInitopt
Expressionopt ForUpdateopt)
Statement from The Java Language
Specification, James Gosling et.
al.
56
Scanner and Parser Generators

• Formal language theory has led to many tools that
  automate the generation of scanners and parsers
  from grammar specifications
• Generally called compiler compilers
• Sample tools
• Accent, ALE, Anagram, Bison, BYACC, Cogencee,
  Coco, Depot4, LEX, FLEX, Happy, Holub, LLGEN,
  PRECC, QUEX, RDP, STYX, VisualParse, YACC
• Java tools
• ANTLR, Beaver, Coco/R, CUP, JavaCC, JFLex,
  JParsec, OpenL, SableCC, SJPT

57
Scanner or Lexer Generators

• Scannar (also called lexer) generators produce
  lexical analysers
• A scannar or lexer is used to perform lexical
  analysis, or the breaking up of an input stream
  into meaningful units, or tokens
• Sample Lexers
• Lex, FLex, JLex, Quex, OOLex, re2c, tclex
• FLEX (Fast LEXical analyser generator) a tool
  for automatically generating a lexer or scanner
  (lex.yy.c) given a lex specification (.l)
• Input file .l
• Output file lex.yy.c

58
FLex Input File

• The general format of FLex input file (.l)
• definitions rules
  subroutines
• Definitions macros and header files
• Rules patterns and associated C statements
• Subroutines C statements and functions

59
Sample Input File

• / int.l input file for the lexer recognizing
  strings of integers in the inputinclude
  ltstdio.hgtoption noyywrap / Tell flex to
  read only one input file / 0-9
  printf(Found an integer s\n", yytext) .
  / Ignore all other characters /int
  main(void) / Call the lexer, then quit
  / yylex() return 0

60
Lexer Production and Usage

• Production
• flex int.l ? lex.yy.c
• gcc o int lex.yy.c ? int
• Usage
• For the input abc123t5!/6yz
• The int lexer produces
• Found an integer 123
• Found an integer 5
• Found an integer 6

61
Parser Generators

• Parser generators produce syntax analysers
• A parser performs syntactic analysis based on a
  formal grammar written in a notation similar to
  BNF
• Sample Parsers
• LLGEN, PRECC, JavaCC, SableCC, YACC, STYX
• YACC (Yet Another Compiler Compiler) a tool for
  automatically generating a parser (y.tab.c) given
  a grammar written in a yacc specification (.y)
  A grammar specifies a set of production rules,
  which define a language, and corresponding
  actions to perform the semantics.
• Input file .y
• Output file y.tab.c

62
YACC Input File

• The same format as FLEX
• definitions rules
  subroutines
• Rule format name names and 'single
  character's
• alternatives

63
Sample Input File(calc.y)
64
Parser Production and Usage

• Production
• yacc calc.y ? y.tab.c
• gcc o calc y.tab.c ? calc
• Usage
• For the input 23512/3
• The calc parser produces
• 13

65
Conclusion

• We use grammars to define programming language
  syntax, both lexical structure and phrase
  structure
• Connection between theory and practice
• Two grammars, two compiler passes
• Parser-generators can write code for those two
  passes automatically from grammars

66
Conclusion, Continued

• Multiple audiences for a grammar
• Novices want to find out what legal programs look
  like
• Expertsadvanced users and language system
  implementerswant an exact, detailed definition
• Toolsparser and scanner generatorswant an
  exact, detailed definition in a particular,
  machine-readable form