6 SYSTEM TEST AND EVALUATION

1
6 SYSTEM TEST AND EVALUATION

• the examination and assessment of a system (or
  an element of a system) in terms of relative
  worth, quality of performance, degree of
  effectiveness, anticipated cost, and so on

2
Evaluation should be a continuous process which
begins during conceptual design, extends through
the operational use and support phase, and
concludes when the system is retired. The purpose
of evaluation is to determine (through a
combination of prediction, analysis, and
measurement activities) true system
charac-teristics and to ensure that the system
will successfully fulfill its intended purpose or
mission.
3
Categories of testing

• The evolution of system evaluation is
  illustrated in Figure 6.2. Various categories of
  testing are identified by program phase, and the
  effectiveness of the evaluation effort increases
  when progressing through types 1, 2, 3, and 4
  testing.

4
(No Transcript)
5
Type 1 Testing

• During the early phases of detail design,
  breadboards, bench-test models, engineering
  models, engineering software, and service test
  models are built with the intent of verifying
  certain performance and physical design
  characteristics. These models, representing
  either an entire system or a designated system
  component, usually operate functionally
  (electrically and mechanically) but do not by any
  means represent production equipment. In the
  development of software, the application of rapid
  prototyping is sometimes used to verify design
  adequacy

6
Type 2 Testing

• Formal tests and demonstrations are accomplished
  during the latter part of detail design when
  preproduction prototype equipment, software, and
  the like are available. Prototype equipment is
  similar to production equipment (that which will
  be deployed for operational use), but it is not
  necessarily fully qualified at this point in
  time. A test program may constitute a series of
  individual tests tailored to the need. Such a
  program might include the following .

7
Type 2 Testing

• 1. Performance tests are accomplished to verify
  individual system performance characteristics.
  For instance, tests are designed to determine
  whether the electric motor will provide the
  necessary output, whether the airplane will
  perform its intended mission successfully, and so
  on. Also, it is necessary to verify form, fit,
  interchangeability, product safety, and other
  comparable features.
• .

8
Type 2 Testing

• 2. Environmental qualification- Temperature
  cycling, shock and vibration, humidity, wind.
  salt spray, dust and sand. fungus, acoustic
  noise, pollution emission, explosion proofing,
  and electromagnetic interference tests are
  conducted. These factors are oriented to what the
  various system elements will be subjected to
  during operation, maintenance, and during
  transpor-tation and handling functions .

9
Type 2 Testing

• 3. Structural tests are conducted to determine
  material characteristics relative to stress,
  strain, fatigue, bending, torsion, and general
  decomposition.
• 4. Reliability qualification are accomplished on
  one or more system elements to determine the MTBF
  and MTBM. Also, special tests are often designed
  to measure component life, to evaluate
  degradation, and to determine modes of failure.

10
Type 2 Testing

• 5. Maintainability demonstration - Tests are
  conducted on one or more system elements to
  assess the values for mean active maintenance
  time (M), mean corrective maintenance time (Mct),
  mean preventive maintenance time (Mpt),
  maintenance labor-hours per operating hour
  (MLH/OH), and so on. In addition maintenance
  tasks, task times and sequences. prime
  equipment-test equipment interfaces, maintenance
  personnel quantities and skills, maintenance
  procedures and maintenance facilities are
  verified to varying degrees.

11
Type 2 Testing

• 6. Support equipment compatibility tests are
  often accomplished to verify compatibility among
  the prime equipment, test and support equipment,
  and transportation and handling equipment.
• 7. Personnel test and evaluation are often
  accomplished to verify the relationships between
  people and equipment, the personnel quantities
  and skill levels required, and training needs.
  Both operator and maintenance tasks are
  evaluated.

12
Type 2 Testing

• 8. Technical data verification - The
  verification of operational and maintenance
  procedures is accomplished.
• 9. Software verification - The verification of
  operational and maintenance software is
  accomplished. This includes computer software
  units (CSUs), computer software configuration
  items (CSCIs), hardware-software compatibility,
  software reliability and maintainability, and
  related testing.

13
Type 2 Testing

• Another aspect of testing in this category
  involves production sampling tests when multiple
  quantities of an item are produced. The tests
  defined earlier basically "qualify" the item
  that is, the equipment hardware configuration
  meets the requirements for production and
  operational use. However, once an item is
  initially qualified, some assurance must be
  provided that all subsequent replicas of that
  item are equally qualified.

14
Type 2 Testing

• Type 2 tests are generally performed in the
  producer or supplier's facility by people at that
  facility. Test and support equipment, designated
  for operational use, and preliminary technical
  manual procedures are employed where possible.
  User personnel often observe and/or participate
  in the testing activities. Design changes as a
  result of corrective action are handled through a
  formalized engineering change procedure.

15
Type 3 Testing

• Formal tests and demonstrations, started after
  initial system qualification and prior to the
  completion of production, are accomplished at a
  designated field test site by user personnel.
  Operational test and support equipment,
  operational spares, and formal operator and
  maintenance procedures are used. Testing is
  generally continuous, accomplished over an
  extended period of time, and covers the
  evaluation of a number of system elements
  scheduled through a series of simulated
  operational exercises.

16
Type 3 Testing

• This is the first time that all elements of the
  system are operated and evaluated on an
  integrated basis. The compatibility of the prime
  equipment with software and the elements of
  maintenance and logistic support is verified as
  well as the compatibility of the various elements
  of support with each other.

17
Type 3 Testing

• Turnaround times and transportation times, stock
  levels, personnel effectiveness factors, and
  other related operational and support parameters
  are measured. In essence, system performance
  (based on certain use conditions) and operational
  readiness characteristics (i.e., operational
  availability, dependability, system
  effectiveness, etc.) can be determined to a
  certain extent.
• .

18
Type 4 Testing

• During the operational-use phase, formal tests
  are sometimes conducted to gain further insight
  in a specific area. It may be desirable to vary
  the mission profile or the system utilization
  rate to determine the impact on total system
  effectiveness, or it might be feasible to
  evaluate several alternative support policies to
  see whether system operational availability can
  be improved. Even though the system is designed
  and operational in the field, this is actually
  the first time that we really know its true
  capability

19
Planning for Test and Evaluation

• For each program, an integrated test and
  evaluation plan is prepared, usually for
  implementation beginning in the preliminary
  design phase. Although the specific content may
  vary somewhat depending on system requirements,
  the plan will generally include the following

20
Planning for Test and Evaluation

• 1. The definition and schedule of all test
  requirements, including anticipated test output
  (in terms of what the test is to accomplish) for
  each individual test and integrated where
  possible. In determining test requirements, some
  components of the system may go through each
  category of the test, whereas other components
  may undergo only a limited amount of testing.
  This is a function of the degree of design
  definition and the risks associated with each
  item in question. An example of this approach is
  illustrated in Figure 6.3.

21
(No Transcript)
22
Planning for Test and Evaluation

• 2. The definition of organization,
  administration, and control responsibilities .
• 3. The definition of test conditions and
  maintenance and logistic resource requirements.
• 4. A description of the test preparation phase
  for each type of testing (selection of specific
  test method, training of test personnel,
  acquisition of logistic resource requirements,
  and preparation of facilities).

23
Planning for Test and Evaluation

• 5. A description of the formal test phase (test
  procedures and test data collection reduction,
  and analysis methods). .
• 6. A description of conditions and provisions
  for a retest phase (methods for conducting
  additional testing as required due to a reject
  situation).
• 7. The identification of test documentation
  (test reporting requirements).

24
Preparation for System Test and Evaluation

• After initial planning, and prior to the start
  of formal evaluation, a period of time is set
  aside for test preparation. During this period,
  the proper conditions must be established to
  ensure effective results. Although there is some
  variance, depending on the type of evaluation,
  these conditions or prerequisites include

25
Preparation for System Test and Evaluation

• the selection of the item(s) to be tested,
• establishment of test procedures,
• test site selection,
• selection and training of test personnel,
• preparation of test facilities and resources,
• the acquisition of support equipment and
• test supply support.

26
Test Performance and Reporting

• With the necessary prerequisites established,
  the next step is to commence with the formal test
  and demonstration of the system. This requires
  operating and supporting the system in a
  prescribed manner as defined in the system test
  and evaluation plan. Throughout this process,
  data are collected and analyzed, which leads to
  assessment of system performance and
  effectiveness characteristics.

27
Test Performance and Reporting

• With the system in an operational status, the
  following questions arise
• What is the true performance and effectiveness of
  the system?
• What is the true effectiveness of the maintenance
  and logistic support capability?
• Are the initially specified requirements being
  met?
• Providing answers to these questions requires a
  formalized data-information and feedback
  capability with the proper output.

28
Test data requirements

• The purpose of a test data and information
  feedback subsystem is twofold.
• It provides ongoing data that are analyzed to
  evaluate and assess the performance,
  effectiveness, operation, maintenance, logistic
  support capability, and so on, for the system in
  the field.
• It provides historical data (covering existing
  systems in the field) that are applicable in the
  design and development of new systems having a
  similar function and nature.

29
Development of a data subsystem

• A format for data collection must be developed,
  and should include both success data and
  maintenance data.
• Success data constitute information covering
  system operation and use on a day-to-day basis.
• Maintenance data cover each event involving
  scheduled and unscheduled maintenance.
• The events are recorded and referenced in system
  operational information reports.

30
System evaluation and corrective action

• The evaluation aspect can address both the
  system as an entity or individual segments of the
  system on an independent basis..
• Corrective action may be accomplished in
  response to a system deficiency, or may be
  accomplished to improve system performance,
  effectiveness, or logistic support. If corrective
  action is to be accomplished, the necessary
  planning and implementation steps are a
  prerequisite to ensure the complete compatibility
  of all elements of the system throughout the
  change process

31
Test reporting

• The final effort in the evaluation process
  constitutes the preparation of an appropriate
  test report. A test report should reference the
  initial system evaluation planning document and
  should describe all test conditions, incorporated
  system modifications during the test (if any),
  test data, and the results of data analysis.
  These results may include appropriate
  recommendations for operation and support of the
  system as applicable to the utilization phase.

32
System Modification

• When a change occurs in a procedure. the prime
  equipment, an element of software, or an item of
  logistic support, the change in most instances
  will affect many different elements of the
  system. Each change must be thoroughly evaluated
  in terms of its impact on other elements of the
  system prior to a decision on whether or not to
  incorporate the change. If a change is to be
  incorporated, the necessary change control
  procedures must be implemented.
• An abbreviated version of a change control
  process is illustrated in Figure 6.9.

33
(No Transcript)