A Method for Verification and Validation Certificate Management in Eclipse

1
A Method for Verification and Validation
Certificate Management in Eclipse

• Mark Sherriff and Laurie Williams
• North Carolina State University
• SoftCeMent 05
• November 8, 2005

2
Agenda

• Background
• Motivation and Hypothesis
• Parametric Modeling in Software Engineering
• Research Methodology
• Building the Model
• Verification Validation Certificate Management
• Parametric model for estimating defect density
• Tool Support
• Limitations
• Current Research Efforts
• Questions

Slide 2 of 17
3
Motivation

• Software Reliability
• Often not estimated until development is complete
• Actual reliability not known until system is
  shipped to customers
• Corrective action is more expensive later in the
  process
• If defect density could be estimated in-process
• Steps could be taken to address issues early
• More economical, could improve development effort

Slide 3 of 17
4
Hypothesis

• Defect density estimation can be based upon the
  history of verification and validation techniques
  that have been performed on the project.

Slide 4 of 17
5
Background - STREW

• Software Testing Reliability Early Warning
• Java version Nagappan Haskell version -
  Sherriff.
• Uses a suite of metrics gathered on static code
  to provide a reliability estimate
• The prediction model is calibrated to an
  organization using a regression equation formed
  from previous metric values
• Why use STREW? Because
• Operational profiles are expensive to create and
  maintain
• An effective automated testing suite could return
  no failures
• We want this method to be able to be used
  in-process to affordably guide corrective action
• Parametric modeling has been an effective tool to
  predict software projects
• Works for testing and static metrics what if we
  add other VV information to the model?

Slide 5 of 17
6
Parametric Modeling

• Method by which dependant variables are related
  to one or more independent variables with regards
  to previous data
• In Software Engineering
• Purpose is to provide an estimated answer to a
  software development question earlier in the
  development lifecycle
• Famous SE parametric models COCOMO 81 and COCOMO
  II

Slide 6 of 17
7
Building the Model
Sherriff, M., Boehm, B., Williams, L., and
Nagappan, N. An Empirical Process for Building
and Validating Software Engineering Parametric
Models. Submitted to Empirical Software
Engineering Journal.
Slide 7 of 17
8
Literature Review and Expert Opinion

• Verification and validation efforts used to
  improve system reliability
• But
• Often this effort is not recorded or managed in a
  systematic way
• Lack of proper documentation can hinder
  development and duplicate effort
• Could be useful information in both project
  management and system maintenance
• Software certificates
• Software certification demonstrates the
  reliability of software systems in such a way
  that it can be checked by an independent
  authority.
• Programatica for Haskell (OGI/OHSU)

Slide 8 of 17
9
Parametric Model and Method

• Defect Estimation with VV Certificates on
  Programming (DevCOP)
• Two main components
• Software Certificate Management System
• DevCOP Certificates and Eclipse Plugin
• Parametric Model for defect density estimation
• Provide guidance to developers on current VV
  efforts

Slide 9 of 17
10
Formulate a-priori model

• DevCOP Certificate
• Includes
• Basic identifying information of creator
• Basic identifying information for code being
  certified
• Record / Evidence of VV technique
• Hash of certified source code
• Certificates follow the code base and can be
  referenced at any time

Slide 10 of 17
11
Certificate Management

• Types of certificates
• Informal
• Includes all manual techniques, such as pair
  programming and code inspections
• Automated Static Analysis
• Includes all techniques that can be run on
  uncompiled code, such as automated static
  analysis
• Dynamic
• Includes all techniques that are run at runtime,
  such as automated testing suites
• Formal
• Includes all formal methods, such as proofs
• Each provides a different type of
  evidence/assurance of reliability

Slide 11 of 17
12
Gather Initial Data

• DevCOP Software Certificate Management System
• Eclipse Plugin v. 1.2
• A method for enabling developers to record
  certificates without excessive overhead
• Active Certificates
• Version Log
• v. 1.0 record basic informal certificates to
  study pair programming vs. code inspections
• v. 1.1 added Active Certificate functionality
• v. 1.1.2 added support for Eclipse rename
  refactoring
• v. 1.2 currently adding coverage statistics

Slide 12 of 17
13
DevCOP Eclipse Plugin Demo

• Available http//agile.csc.ncsu.edu/mssherri/devc
  op/

Slide 13 of 17
14
Limitations

• Granularity of Certificates
• Method level, not class or line of code
• Composition of Certificates
• Not much is known about how one VV technique
  adds to another
• Defect Severity
• All defects are treated equally
• Certificate rubric specificity
• Making relative effectiveness scale is hard

Slide 14 of 17
15
Validation

• Base Model with Customization
• Take model and calibrate it to projects
• from different companies / development teams
• of different scopes
• with different programming languages
• with different VV styles
• Show that model can be effective estimator in
  each scenario

Slide 15 of 17
16
Potential Side Effects

• Retrospective Causal Analysis
• Once certificates are recorded on a project and
  bugs are reported, developers can use certificate
  and defect information to evaluate the efficacy
  of their VV practices.
• Building certificate information in with compiled
  code base
• If certificate information could travel with
  compiled code, it could be referenced at runtime
  so that other systems could evaluate whether it
  wants to work with that system.

Slide 16 of 17
17
Thank you!

• Questions? Queries? Quandaries?
• Contact Information
• Mark Sherriff
• mark.sherriff_at_ncsu.edu
• http//www4.ncsu.edu/mssherri/
• http//research.csc.ncsu.edu/softwareengineering/
  realsearch/
• DevCOP Plugin http//agile.csc.ncsu.edu/mssherri/
  devcop/

Slide 17 of 17