Overview of LEX and YACC - Department of Computer Engineering

1
Systems Programming Operating
SystemsUnit III
International Institute of Information
Technology, Pune Department of Computer
Engineering

• Case Study Overview of LEX and YACC
• Prof. Deptii Chaudhari
• Assistant Professor
• Department of Computer Engineering

2
LEX YACC

• What is Lex?
• Lex is officially known as a "Lexical Analyser".
• It's main job is to break up an input stream into
  more usable elements.
• Or in, other words, to identify the "interesting
  bits" in a text file.
• What is Yacc?
• Yacc is officially known as a "parser".
• In the course of it's normal work, the parser
  also verifies that the input is syntactically
  sound.
• YACC stands for "Yet Another Compiler Compiler".
  This is because this kind of analysis of text
  files is normally associated with writing
  compilers.

Deptii Chaudhari, Dept of Computer Engineering,
Hope Foundations International Institute of
Information Technology, I²IT P-14,Rajiv Gandhi
Infotech Park MIDC Phase 1, Hinjawadi, Pune
411057 Tel - 91 20 22933441/2/3
www.isquareit.edu.in info_at_isquareit.edu.in
3
Deptii Chaudhari, Dept of Computer Engineering,
Hope Foundations International Institute of
Information Technology, I²IT P-14,Rajiv Gandhi
Infotech Park MIDC Phase 1, Hinjawadi, Pune
411057 Tel - 91 20 22933441/2/3
www.isquareit.edu.in info_at_isquareit.edu.in
4
LEX Program Structure
C global variables, prototype, Comments
Definitions
Production Rules
------------------------------------
User Subroutine Section (Optional)
Deptii Chaudhari, Dept of Computer Engineering,
Hope Foundations International Institute of
Information Technology, I²IT P-14,Rajiv Gandhi
Infotech Park MIDC Phase 1, Hinjawadi, Pune
411057 Tel - 91 20 22933441/2/3
www.isquareit.edu.in info_at_isquareit.edu.in
5

• In the rules section, each rule is made up of two
  parts a pattern and an action separated by
  whitespace.
• The lexer that lex generates will execute the
  action when it recognizes the pattern.
• The user subroutine section, consists of any
  legal C code.
• Lex copies it to the C file after the end of the
  lex generated code.
• Lex translates the Lex specification into C
  source file called lex.yy.c which we compile and
  link with lex library ll.
• Then we can execute the resulting program to
  check that it works as we expected.

Deptii Chaudhari, Dept of Computer Engineering,
Hope Foundations International Institute of
Information Technology, I²IT P-14,Rajiv Gandhi
Infotech Park MIDC Phase 1, Hinjawadi, Pune
411057 Tel - 91 20 22933441/2/3
www.isquareit.edu.in info_at_isquareit.edu.in
6
Example

• include ltstdio.hgt
• 0123456789 printf("NUMBER\n")
• a-zA-Za-zA-Z0-9 printf("WORD\n")
• Running the Program
• lex example_lex.l
• gcc lex.yy.c ll
• ./a.out

Deptii Chaudhari, Dept of Computer Engineering,
Hope Foundations International Institute of
Information Technology, I²IT P-14,Rajiv Gandhi
Infotech Park MIDC Phase 1, Hinjawadi, Pune
411057 Tel - 91 20 22933441/2/3
www.isquareit.edu.in info_at_isquareit.edu.in
7
Pattern Matching Primitives
Metacharacter Matches
. any character except newline
\n newline
zero or more copies of the preceding expression
one or more copies of the preceding expression
? zero or one copy of the preceding expression
beginning of line
end of line
ab a or b
(ab) one or more copies of ab (grouping)
"ab" literal "ab" (C escapes still work)
character class
Deptii Chaudhari, Dept of Computer Engineering,
Hope Foundations International Institute of
Information Technology, I²IT P-14,Rajiv Gandhi
Infotech Park MIDC Phase 1, Hinjawadi, Pune
411057 Tel - 91 20 22933441/2/3
www.isquareit.edu.in info_at_isquareit.edu.in
8
Pattern Matching Examples
Expression Matches
abc abc
abc ab abc abcc abccc ...
abc abc, abcc, abccc, abcccc, ...
a(bc) abc, abcbc, abcbcbc, ...
a(bc)? a, abc
abc one of a, b, c
a-z any letter, a through z
a\-z one of a, -, z
-az one of - a z
A-Za-z0-9 one or more alphanumeric characters
\t\n whitespace
ab anything except a, b
ab a, , b
ab a, , b
ab a, b
Deptii Chaudhari, Dept of Computer Engineering,
Hope Foundations International Institute of
Information Technology, I²IT P-14,Rajiv Gandhi
Infotech Park MIDC Phase 1, Hinjawadi, Pune
411057 Tel - 91 20 22933441/2/3
www.isquareit.edu.in info_at_isquareit.edu.in
9
Operation of yylex()

• When lex compiles the input specification, it
  generates the C file lex.yy.c that contains the
  routine int yylex(void).
• This routine reads the input string trying to
  match it with any of the token patterns specified
  in the rules section.
• On a match associated action is executed.
• When we call yylex() function, it starts the
  process of pattern matching.
• Lex keeps the matched string into the address
  pointed by pointer yytext.
• Matched string's length is kept in yyleng while
  value of token is kept in variable yylval.

Deptii Chaudhari, Dept of Computer Engineering,
Hope Foundations International Institute of
Information Technology, I²IT P-14,Rajiv Gandhi
Infotech Park MIDC Phase 1, Hinjawadi, Pune
411057 Tel - 91 20 22933441/2/3
www.isquareit.edu.in info_at_isquareit.edu.in
10
int com0 "/"\n"/"
comfprintf(yyout, " ") int
main() printf("Write a C program\n") yyoutfope
n("output", "w") yylex() printf("Commentd\n",c
om) return 0
cc lex.yy.c -ll ./a.out Write a C
program includeltstdio.hgt int main() int a,
b /float c/ printf(Hi)
/printf(Hello)/ Comment2 cat
output includeltstdio.hgt int main() int a,
b printf(Hi)
Deptii Chaudhari, Dept of Computer Engineering,
Hope Foundations International Institute of
Information Technology, I²IT P-14,Rajiv Gandhi
Infotech Park MIDC Phase 1, Hinjawadi, Pune
411057 Tel - 91 20 22933441/2/3
www.isquareit.edu.in info_at_isquareit.edu.in
11
Lex Predefined Variables
Deptii Chaudhari, Dept of Computer Engineering,
Hope Foundations International Institute of
Information Technology, I²IT P-14,Rajiv Gandhi
Infotech Park MIDC Phase 1, Hinjawadi, Pune
411057 Tel - 91 20 22933441/2/3
www.isquareit.edu.in info_at_isquareit.edu.in
12
YACC

• YACC is a parser generator that takes an input
  file with an attribute-enriched BNF (Backus
  Naur Form) grammar specification.
• It generates the output C file y.tab.c containing
  the function int yyparse(void) that implements
  its parser.
• This function automatically invokes yylex()
  everytime it needs a token to continue parsing.

Deptii Chaudhari, Dept of Computer Engineering,
Hope Foundations International Institute of
Information Technology, I²IT P-14,Rajiv Gandhi
Infotech Park MIDC Phase 1, Hinjawadi, Pune
411057 Tel - 91 20 22933441/2/3
www.isquareit.edu.in info_at_isquareit.edu.in
13
Deptii Chaudhari, Dept of Computer Engineering,
Hope Foundations International Institute of
Information Technology, I²IT P-14,Rajiv Gandhi
Infotech Park MIDC Phase 1, Hinjawadi, Pune
411057 Tel - 91 20 22933441/2/3
www.isquareit.edu.in info_at_isquareit.edu.in
14
Structure of YACC Program
C global variables, prototype, Comments
Definitions
Context free grammar action for each production
------------------------------------
Subroutines/Functions
Deptii Chaudhari, Dept of Computer Engineering,
Hope Foundations International Institute of
Information Technology, I²IT P-14,Rajiv Gandhi
Infotech Park MIDC Phase 1, Hinjawadi, Pune
411057 Tel - 91 20 22933441/2/3
www.isquareit.edu.in info_at_isquareit.edu.in
15
Arithmatic.l includeltstdio.hgtinclude
"y.tab.h"extern int yylval0-9
          yylvalatoi(yytext)          return
NUMBER       \t \n return 0. return
yytext0int yywrap()return 1
How To Run yacc -d arithmatic.y lex
arithmatic.l gcc lex.yy.c y.tab.c ./a.out
Deptii Chaudhari, Dept of Computer Engineering,
Hope Foundations International Institute of
Information Technology, I²IT P-14,Rajiv Gandhi
Infotech Park MIDC Phase 1, Hinjawadi, Pune
411057 Tel - 91 20 22933441/2/3
www.isquareit.edu.in info_at_isquareit.edu.in
16
References

• https//www.epaperpress.com/lexandyacc/
• John. R. Levine, Tony Mason and Doug Brown - Lex
  and Yacc?, O'Reilly

Deptii Chaudhari, Dept of Computer Engineering,
Hope Foundations International Institute of
Information Technology, I²IT P-14,Rajiv Gandhi
Infotech Park MIDC Phase 1, Hinjawadi, Pune
411057 Tel - 91 20 22933441/2/3
www.isquareit.edu.in info_at_isquareit.edu.in
17

• THANK YOU
• For further details, please contact
• Deptii Chaudhari
• deptiic_at_isquareit.edu.in
• Department of Computer Engineering
• Hope Foundations
• International Institute of Information
  Technology, I²IT
• P-14,Rajiv Gandhi Infotech Park
• MIDC Phase 1, Hinjawadi, Pune 411057
• Tel - 91 20 22933441/2/3
• www.isquareit.edu.in info_at_isquareit.edu.in