SSS Testing and Validation

1
SSS Testing and Validation

• February 23, 2003
• Chicago, IL
• Neil Pundit
• Erik DeBenedictis
• William McLendon III

Sandia is a multiprogram laboratory operated by
Sandia Corporation, a Lockheed Martin
Company,for the United States Department of
Energy under contract DE-AC04-94AL85000.
2
Overview

• Strategies for RT System Testing and Validation
• QMTest
• Application Test Packages
• APITEST
• Cluster Integration Toolkit

3
Distributed Runtime System Testing

• Complicated interactions
• Approach to testing
• Component Testing
• Light Applications
• Performance / Functionality
• Operational Profile
• Stress Testing
• Users expect a high-degree of quality in todays
  high end systems! Extensive (and smart) testing
  is critical to achieving this goal.

4
Review

• Review of topics covered at previous F2F meetings.

5
QM Test Review

• Scriptable test driver (Python gt 2.1)
• Test Metadata defined with XML
• HTTP interface (Zope)
• HTML are customizable
• Open source
• Active development community
• URL
• http//www.codesourcery.com/

6
QM Test Screenshot
7
Application Test Framework Review

• Presented by Mike Carifio at November meeting
• Test Framework Packages
• Package driver GNU Make
• Easy to use interface
• make targets!
• Archival of results
• Used on production CPlant systems

8
Application Test Packages

• Templates for Packages available
• Easy to add new packages
• Some Packages Already Included
• Linpack
• NAS-Parallel Benchmarks
• BLACS
• MPI-Test

9
APITEST
10
APITEST - Overview

• Unit-testing tool for network components
• Idea from October 2002 meeting
• Scriptable Tests (XML Schema Grammar)
• Enabling Technology
• Black/White Box Testing
• Multi-Protocol Support
• TCP/IP, SSSLib, Portals

11
APITEST Feature list

• Scriptable Tests
• XML Schema
• Test Entity Types
• Command Shell
• Network Transmission
• Test Dependencies
• Parent
• Sibling

• Distributed Test Execution
• Test Repetition
• Delayed loops
• Multiple Protocols
• TCP/IP (SSSLib)
• Portals
• Result Handling
• Historical Data
• Archival

12
APITEST Concept
13
APITEST Enables

• Flexibility in unit testing without a lot of
  coding
• Black Box testing
• Does a component support the API?
• Fault injection?
• RT Integration testing
• Component abstraction
• Path analysis
• White Box testing
• Coverage Testing
• Internal States of Component X
• Unreachable States?

14
APITEST Example Script
15
APITEST Summary

• New effort
• Concept from October 2001 F2F discussions.
• Good progress so far
• Developed set of features and capabilities
• Developed XML Schema definition for test entities
• Implementation of v0.1a under way
• C
• Xerces for XML parsing and Schema validation
• Data structures 75 implemented at this time.

16
Cluster Integration Toolkit (CIT)
17
Cluster Integration Toolkit

• Used for many integration and management tasks on
  Cplant
• Numerous cluster installs
• Demonstrated scalability
• Largest 1861 nodes.
• daemon interfaces adaptable to XML over IP
• command_daemon
• status_daemon
• James H. Laros III (jhlaros_at_sandia.gov)

18
Example status_daemon Config File
lt?xml version"1.0" ?gt ltconfiggt
ltportgt2009lt/portgt ltpid_filegt/tmp/csdaemon.pidlt/
pid_filegt ltlog_levelgt4lt/log_levelgt
ltsd_local_checksgtonlt/sd_local_checksgt
ltsd_check_delaygt30lt/sd_check_delaygt
ltsd_report_delaygt300lt/sd_report_delaygt
ltsd_conn_timeoutgt10lt/sd_conn_timeoutgt
ltsd_parent_alert_retriesgt3lt/sd_parent_alert_retrie
sgt ltsd_admin_log_dirgt/tmplt/sd_admin_log_dirgt
ltsd_trendinggtofflt/sd_trendinggt
ltbackgroundgt0lt/backgroundgt lt/configgt
19
status_daemon Example Load Avg.
20
What CIToolkit offer to SciDAC

• How much work to integrate an interface to allow
  these daemons to speak SciDAC?
• Good integration test
• SSS XML Interface
• Will be a good exercise

21
Conclusions

• Testing and Validation
• Framework for testing Distributed RT system.
• Existing capabilities
• Application Testing
• Stress Tests
• New capabilities...
• APITEST
• Quite a lot of work still to do
• CIToolkit integration with SciDAC?