Specification and Reasoning in SE Projects Using a Web IDE

1
Specification and Reasoning in SE Projects
Using a Web IDE

• Charles T. Cook (Clemson)
• Svetlana V. Drachova-Strang (Limestone College)
• Yu-Shan Sun (Clemson)
• Murali Sitaraman (Clemson)
• Jeffrey C. Carver (Alabama)
• Joseph E. Hollingsworth (IU Southeast)
• This research is funded in part by NSF grants
  CCF-0811748, CCF-1161916, DUE-1022191, and
  DUE-1022941.

2
Part I Overview
3
About Clemson

• School of Computing has about 600 undergrads and
  200 grads
• Clemson University has 17,000 students
• Located in a town (also Clemson) with an official
  population of about 13,000 in South Carolina
• Has a football stadium with capacity 85,000

4
This talk

• This talk is about undergrad courses, though the
  ideas have been used in both undergrad and grad
  courses for many years

5
Goals of the SE Projects

• Role of formal specifications as contracts in
  team software development and integration
• Role of formal specifications in reasoning about
  software correctness
• Other uses
• Specification-based test case design

6
Intro. and Advanced Projects

• Introductory projects involve only use of and
  reasoning with existing library components
• 2 weeks of lectures
• Advanced projects involve development and
  reasoning about new components
• 3-5 weeks of lectures
• Other variations

7
Sample Courses and Institutions

• Sample introductory projects
• Alabama (Software Engineering)
• Cleveland State (Software Engineering)
• Sample advanced projects
• Clemson (Software Engineering)
• Denison (Independent study projects)
• Other variations
• Depauw (Theory), NC State (Data structures),
  Southern Wesleyan (Data Structures), Ramapo
  College (Programming Languages),

8
Clemson University Details

• Two-course sequence
• Sophomore-Level, CP SC 215 Software Development
  Foundations
• Junior/Senior-Level CP SC 372 Software
  Engineering
• Experimentation and assessment
• Piloting (2007/2008)
• Institutionalization (2009 to present)

9
Soft. Dev. Foundations Course

• Intro to Java, object-based computing, software
  engineering, design patterns, etc.
• 10 weeks (includes usual materials and projects)
• Intro to formal specifications and reasoning
  (interspersed with above topics)
• 4 weeks
• Uses RESOLVE-style specifications in a Java
  context

10
Software Engineering Course

• Software life cycle, process models, requirements
  analysis and design
• 8 weeks (includes usual materials and projects)
• Transition from informal to formal
• 1 week
• Specification-based component development and
  quality assurance
• 5 weeks
• Uses RESOLVE

11
Clemson University Key Points

• Specification/reasoning included in syllabi for
  two required courses for majors (2009)
• Graduating student learning outcome modified to
  include development of bug-free software
  according to specifications (2012)

12
Part II Example Projects
13
RESOLVE

• An integrated specification and programming
  language for verified, component-based software
  development
• A Verifying compiler
• A github project
• A freely-available web interface to use the
  verifying compiler
• www.cs.clemson.edu/group/resolve

14
Sample Intro Assignments

• Generate VCs and prove the given Queue
  Remove_Last operation.
• Make each of the following changes and explain
  what is unprovable.
• Comment out the first Dequeue operation.
• Change the maintaining clause (loop invariant) to
  Q ltEgt o Q.
• Change the decreasing clause (termination
  progress metric) to T.

15
Sample Advanced Assignments

• Implement and verify
• Extension operations on Queues to Insert_After
  and Remove_After
• Sequence_Template using Queue_Template with
  extensions
• Multiple implementations of Queue_Template
  satisfying given internal contracts
• Assignments to put it all together
• Involve a dozen components and teams of 3 students

16
Sample Components

• Stack_Template
• Queue_Template
• Preemptable_Queue_Template
• Sequence_Template
• List_Template
• Search_Store_Template
• Map_Template
• Prioritizer_Template

17
Part III A Web IDE Demo
18
Getting Started

• www.cs.clemson.edu/group/resolve
• Tab Web IDE
• Google
• RESOLVE web IDE
• RESOLVE verifier
• Clemson RESOLVE

19
Sample Intro Assignments

• Generate VCs and prove the given Queue
  Remove_Last operation.
• Make each of the following changes and explain
  what is unprovable.
• Comment out the first Dequeue operation.
• Change the maintaining clause (loop invariant) to
  Q ltEgt o Q.
• Change the decreasing clause (termination
  progress metric) to T.

20
Elements of the Assignment

• Queue_Template concept specification
• A specification of Remove_Last Operation
• An annotated implementation of Remove_Last
  operation

21
Mathematical Modeling

• Concepts provide mathematical models for
  programming objects
• To write formal specifications, we need to model
  the state mathematically
• Some objects we use in programming, such as
  Integers and Reals, have implicit models
• For others, such as stacks, queues, lists, etc.,
  we need to conceive explicit mathematical models

22
Mathematical Modeling of Queues

• Concept Queue_Template(type Entry Max_Length
  Integer)
• uses String_Theory
• Type Family Queue is modeled by
• Operation Enqueue
• Operation Dequeue
• end Queue_Template

23
Mathematical Modeling of Queues

• Concept Queue_Template(type Entry Max_Length
  Integer)
• uses String_Theory
• Type Family Queue is modeled by
• Str(Entry)
• exemplar Q
• constraints Q lt Max_Length
• initialization ensures Q ?
• end Queue_Template

24
A Specification of Remove_Last

• Operation Remove_Last (updates Q Queue
  replaces E Entry)
• requires Q / 0
• ensures Q Q o ltEgt

25
An Implementation of Remove_Last

• Procedure Remove_Last (updates Q Queue
  replaces E Entry)
• Var T Queue
• Dequeue (E, Q)
• While (Length(Q) / 0)
• do
• Enqueue(E,T)
• Dequeue(E,Q)
• end
• Q T
• end Remove_Last

26
Understanding Loop Invariant

• Assume Q lt10, 20, 30, 40gt
• Dequeue (E, Q)
• While (Length(Q) / 0)
• T E Q
• Iter. 1 ? 10 lt20, 30, 40gt Iter. 2 lt10gt 20 lt
  30, 40gt
• Iter. 3 lt10, 20gt 30 lt40gt
• Iter. 4 lt10, 20, 30gt 40 ?
• do
• Enqueue(E,T)
• Dequeue(E,Q)
• end

27
Understanding Loop Invariant

• Assume Q lt10, 20, 30, 40gt
• Dequeue (E, Q)
• While (Length(Q) / 0)
• T E Q
• Iter. 1 ? 10 lt20, 30, 40gt Iter. 2 lt10gt 20 lt
  30, 40gt
• Iter. 3 lt10, 20gt 30 lt40gt
• Iter. 4 lt10, 20, 30gt 40 ?
• Loop maintains this invariant
• Q T o ltEgt o Q

28
An Implementation of Remove_Last

• Procedure Remove_Last (updates Q Queue
  replaces E Entry)
• Var T Queue
• Dequeue (E, Q)
• While (Length(Q) / 0)
• maintaining Q T o ltEgt o Q
• decreasing Q
• do
• Enqueue(E,T)
• Dequeue(E,Q)
• end

29
Verification

• Press Verify Button
• Generate and prove automatically a series of
  verification conditions (VCs)
• Students understand why the VCs arise
• Understand connections between contracts, code,
  and proofs
• Can prove VCs

30
Impact of Incorrect Code

• The first statement, call to Dequeue, is removed.
• Leads to one unprovable VC (Verification
  Condition)
• VC 0_1
• Base Case of the Invariant of While Statement in
  Procedure Remove_Last Remove_Last_Realiz.rb(8) 
• Goal Q ((empty_string o ltEgt) o Q)
• Givens

31
Part IV Assessment
32
Assessment Summary

• For details
• See the paper
• See Drachova Ph. D. dissertation (2013, Clemson),
  available at website
• A quick summary
• Likert items to assess perceived benefits of web
  IDE (avg. 4.0/5.0)
• Project grades (avg. from 80) about the same as
  non-formal parts
• RCI-item based analysis

33
What reasoning skills are necessary?Reasoning
Concept Inventory
Boolean Logic Standard Logic Symbols, Standard Proof Techniques
Discrete Math Structures Sets, Strings, Numbers, Relations, and other mathematical theories as needed
Precise Specifications Mathematical Descriptions of Software interfaces for clients and implementers. Math models for structures. Pre and Post conditions for operations.
Modular Reasoning Each Module needs to be proven correct only once.
Correctness Proofs Mathematical Assertions equivalent to the correctness of the program. Application of Proof Techniques to the program
http//www.cs.clemson.edu/resolve/teaching/invento
ry.html
34
Clemson RCI-Based Evaluation
  Sem Avg. scoring 70 or higher
RCI 3.4.3 3 82 87 85
RCI 4.2 2 56 76 59
RCI 4.3.1 1 24 77 67

• 3.4.3 Precise specifications
• 4.2 Design by contract
• 4.3.1 Internal contracts

35
SE Project Benefits on RCI-Based Learning Outcomes
  Avg. scoring 70 or higher
Before project (mid-term) 59 25
After project (final) 84 75

• RCI Topics 3.4.2, 3.4.3 Precise specifications
• Sample size 24 students, Spring 12

36
Summary

• Students can practice formal specification and
  reasoning principles effectively using the
  RESOLVE web IDE and its verifier
• There are some benefits for even a minimal intro
  (3 lectures plus project)
• Significant benefits with longer-term exposure
• IDE and materials online
• www.cs.clemson.edu/group/resolve