Client and Server Verification for Web Services Using Interface Grammars

1
Client and Server Verification for Web Services
Using Interface Grammars

• Graham Huges, Tevfik Bultan, Muath Alkhalaf
• Department of Computer Science
• University of California, Santa Barbara

2
Outline

• Motivation
• Interface Grammars
• Interface Compiler
• Interface Grammars for Web Services
• A Case Study
• Conclusions

3
Model Checking Software

• Model checking
• An automated software verification technique
• Exhaustive exploration of the state space of a
  program to find bugs
• Systematically explore all possible behaviors of
  a program
• look for violations of the properties of interest
• assertion violations, deadlock
• Software model checkers Verisoft, Java
  PathFinder (JPF), SLAM, BLAST, CBMC

4
Two Challenges in Model Checking

• State space explosion
• Exponential increase in the state space with
  increasing number of variables and threads
• State space includes everything threads,
  variables, control stack, heap
• Environment generation
• Finding models for parts of software that are
• either not available for analysis, or
• are outside the scope of the model checker

5
Modular Verification

• Modularity is key to scalability of any
  verification technique
• Moreover, it can help isolating the behavior you
  wish to focus on, removing the parts that are
  beyond the scope of your verification technique
• Modularity is also a key concept for successful
  software design
• The question is finding effective ways of
  exploiting the modularity in software during
  verification

6
Interfaces for Modularity

• How do we do modular verification?
• Divide the software to a set of modules
• Check each module in isolation
• How do we isolate a module during
  verification/testing?
• Provide stubs representing other modules
  (environment)
• How do we get the stubs representing other
  modules?
• Write interfaces
• Interfaces specify the behavior of a module from
  the viewpoint of other modules
• Generate stubs from the interfaces

7
Our Approach Interface Grammars

• Here is the basic use case for our approach
• User writes an interface grammar specifying the
  interface of a component
• Our interface compiler automatically generates a
  stub from the interface grammar
• Automatically generated stub provides the
  environment during modular verification

8
Interface Grammars
Interface Compiler
Interface Grammar
Component B
Interface Grammar
Component B Stub
Component A
Model Checker
Component A
9
An Example

• An interface grammar for transactions
• Specifies the appropriate ordering for calls to a
  transaction manager
• Calls are the terminal symbols of the interface
  grammar

Start ? Base Base ? begin Tail Base
e Tail ? commit rollback
10
An Example

• Consider the call sequence
• begin rollback begin commit
• Here is a derivation
• Start
• Base
• begin Tail Base
• begin rollback Base
• begin rollback begin Tail Base
• begin rollback begin commit Base
• begin rollback begin commit

Start ? Base Base ? begin Tail Base
e Tail ? commit rollback
11
Another Example

• This example can also be specified as a Finite
  State Machine (FSM)
• However, the following grammar which specifies
  nested transactions cannot be specified as a FSM

begin
commit rollback
Start ? Base Base ? begin Base Tail Base
e Tail ? commit rollback
12
Yet Another Example

• Lets add another operation called
  setrollbackonly which forces all the pending
  transactions to finish with rollback instead of
  commit
• We achieve this by extending the interface
  grammars with semantic predicates and semantic
  actions

Start ? rfalse l0 Base Base ? begin
ll1 Base Tail ll-1 if l0 then
rfalse Base setrollbackonly rtrue
Base e Tail ? rfalse commit rollback
13
Interface Grammar Translation

• Our interface compiler translates interface
  grammars to executable code
• the generated code is the stub for the component
• The generated code is a parser that
• parses the incoming calls
• while making sure that the incoming calls conform
  to the interface grammar specification

14
Verification with Interface Grammars
Interface Compiler
Interface Grammar
Top-down parser
Program
parser stack
invocation (lookahead)
Component Stub
parse table
semantic predicates and semantic actions
Model Checker
15
Interface Grammars Client vs. Server

• Interface grammars can be used for
• Client verification Generate a stub for the
  server
• Server verification Generate a driver for the
  server

Parser
Client
Stub
Interface Compiler
Interface
Server
Driver
Sentence Generator
16
Interface Grammars and Data

• A crucial part of the interface specification is
  specifying the allowable values for the arguments
  and generating allowable return values
• Approach 1 These can be specified in the
  semantic actions and semantic predicates of the
  interface grammars
• Approach 2 Can we specify the constraints about
  the arguments and return values using the grammar
  rules?
• Yes, grammar productions can be used to specify
  the structure of most recursive data structures.

17
Modular Verification of Web Services

• We applied our modular verification approach
  based on interface grammars to client and server
  side verification of Web services

18
Interface Grammars for Web Services

• Our approach
• A WSDL-to-interface grammar translator
  automatically generates grammar productions that
  generate and/or validate XML arguments and return
  values
• User adds control flow constraints by modifying
  the grammar
• Interface compiler automatically generates a stub
  for client side verification and a driver for
  server-side verification

19
Interface Grammars for Web Services
20
A Case Study AWS-ECS

• We performed both client and server side
  verification for the Amazon E-Commerce Web
  Service (AWS-ECS) using our approach
• AWS-ECS WSDL specification lists 40 operations
• that provide differing ways of searching Amazons
  product database
• We focused on the following core operations
• ItemSearch, CartCreate, CartAdd, CartModify,
  CartGet, CartClear

21
Client Verification

• For client verification we used a demonstration
  client provided by Amazon AWS-ECS Java Sample
• This client does not check any constraints such
  as
• You should not try to insert an item to a
  shopping cart before creating a shopping cart
• When such requests are sent to AWS-ECS they would
  return an error message
• Using our approach we can easily check if the
  client allows such erroneous requests
• We used the Java PathFinder (JPF) model checker
• Falsification time changes with the type of
  faults we are looking for (data or control
  errors), changes from 10 to 60 seconds

22
Client Verification Setup
Web Service Client
Test Driver
Server Stub
Hand written Driver
Automatically Generated Stub/Parser
AWS-ECS Java Sample

• The AWS-ECS Client uses Apache Axis SOAP library
• Given a WSDL specification, Apache Axis
  automatically generates Java classes as wrappers
  for SOAP calls
• We replaced the Java classes generated by Apache
  Axis and forwarded them to our automatically
  generated parser/server stub
• We wrote a driver which non-deterministically
  generates input events for the Client, simulating
  user behavior
• JPF model checker exhaustively explores all
  non-deterministic choices (both in the driver and
  the stub)

23
Input Validation Check

• We checked if the client performs input
  validation upon receiving an input from the user
  before passing the corresponding request to the
  server
• Possible erroneous inputs by the user
• Type errors
• For example Entering a string for a numeric
  field
• Nonsensical data
• For example Adding a nonexistent item to a
  nonexisting shopping cart
• Uncorrelated data
• For example Search for an item but then try to
  insert another nonexisting one

24
Input Validation Experiments with JPF
Type Time (sec.) Memory (MB)
Type error 12.5 25
Nonsensical data 11.1 25
Uncorrelated data 20.8 43
25
Control-Flow Validation Check

• Call sequences that can lead to errors
• Such as modifying the contents of a cart after
  clearing it
• We wrote a driver that
• First initializes the cart and
• Then generates a sequence of user events
  non-deterministically
• We experimented for different event sequence
  sizes

26
Control-Flow Validation Experiments
Type Depth Time (sec.) Memory (MB) Errors
First error 2 31.8 43 0
First error 3 64.2 62 1
First error 4 49.6 73 1
First error 5 57.3 73 1
All errors 2 31.8 43 0
All errors 3 77.3 62 2
All errors 4 266.8 112 15
All errors 5 862.6 230 68
27
AWS-ECS Server Verification

• Our interface compiler automatically generates a
  driver that sends sequences of requests to
  AWS-ECS server and checks that the return values
  conform to the interface specification
• The driver is a sentence generator
• It generates sequences of SOAP requests based on
  the interface specification
• We used two algorithms for sentence generation
• A random sentence generation algorithm
• Purdoms algorithm A directed sentence
  generation algorithm that guarantees production
  coverage

28
Directed Sentence Generation

• Number of sentences generated 5
• Average derivation length 24
• Average number of SOAP requests/responses 3.8
• Verification time 20 seconds

29
Random Sentence Algorithm

• Number of sentences generated 100
• Average derivation length 17.5
• Average number of SOAP requests/responses 3.2

30
Server verification

• We found two errors during server side
  verification
• Errors were discovered within 10 seconds
• These errors indicate a mismatch between the
  interface specification and the server
  implementation
• It may mean that we misunderstood the description
  of the Web service interface
• It may also mean that there is an error in the
  service implementation

31
Conclusions

• Modular verification is a necessity
• Interfaces are crucial for modular verification
• Interface grammars provide a new specification
  mechanism for interfaces
• Interface grammars can be used for automated stub
  generation leading to modular verification for
  both client and server for verification of Web
  Services

32
THE END
33
Related Work Interfaces

• L. de Alfaro and T. A. Henzinger. Interface
  automata.
• O. Tkachuk, M. B. Dwyer, and C. Pasareanu.
  Automated environment generation for software
  model checking.
• A. Betin-Can and T. Bultan. Verifiable
  concurrent programming using concurrency
  controllers.
• T. Ball and S. K. Rajamani. SLAM interface
  specification language.
• G. T. Leavens et al. JML

34
Related Grammar-based Testing

• A. G. Duncan, J. S. Hurchinson Using attributed
  grammars to test designs and implementations
• P. M. Maurer Generating test data with enhanced
  context free grammars
• P. M. Maurer The design and implementation of a
  grammar-based data generator
• E. G. Sirer and B. N. Bershad Using production
  grammars in software testing