EMIS 7307

1
Test Planning and Analysis
Landing distance of an F-18
2
Test Planning and Analysis
3
Test Planning and Analysis
4
Test Planning and Analysis

• What if 60 isnt sufficient confidence?
• Typically 95 is the desired confidence.
• Need more data points!
• How many?
• Cant say for sure.
• Depends on s and x with the new data.

5
Test Planning and Analysis
Lets assume that if we collect more data the
standard deviation and (x -V) dont change. This
not a very good assumption, but possibly the best
we can do. Then we can rewrite ts as ts k
v(n-1). With this assumption k .351. For n
2, ts .351, a.39, (1-a) .61 n
5, ts .702, a.27, (1-a) .73 n
10, ts 1.053, a.16 (1-a) .84 n
17, ts 1.404, a.09 (1-a) .91 n 26,
ts 1.755, a.04 (1-a) .96 n 37,
ts 2.106, a.015 (1-a) .985
6
Test Planning and Analysis

• From the previous chart we get a sense of the
  cost of confidence.
• For an increase of only three tests (from 2 to 5)
  we can boost our confidence by 12.
• But the change from 17 to 26 only further
  increased our confidence by 5!
• This illustrates why 90 confidence is often
  chosen as good enough. Its a compromise of
  increased confidence versus cost.

7
Test Planning and Analysis
ANOVA
8
Test Planning and Analysis
9
Test Planning and Analysis
ANOVA
10
Test Planning and Analysis
11
Test Planning and Analysis
12
Test Planning and Analysis
13
Test Planning and Analysis
14
Test Planning and Analysis
15
Test Planning and Analysis

• In this example the table f value is 2.78. Use 4
  and 25 as the degrees of freedom.
• The calculated f is 4.3.
• Therefore, reject the null hypothesis.

16
Test Planning and Analysis
17
Test Planning and Analysis

• ANOVA is a powerful data analysis tool
• Using a few samples answers the question, does an
  effect on a subgroup of a hypothetical population
  cause that subgroup to not really be from the
  hypothetical population?
• We hypothesized that the five different
  aggregates were from a population that absorbed
  moisture similarly.
• The ANOVA showed that they were not!

18
Test Planning and Analysis

• ANOVA is readily available in many spreadsheet
  packages.
• Excel has it under tools then click on data
  analysis and select ANOVA.

19
Test Planning and Analysis

• Bottom line.
• Use student t or normal distribution (ngt30) when
  comparing to a spec.
• Use ANOVA when looking to see if two groups are
  statistically the same to some confidence level.

20
Test Planning and Analysis

• Bottom line (contd)
• For both test planning and data analysis at least
  one member of the team needs to be statistics
  savvy.
• From a management perspective.
• This facilitates planning for adequate data
  collection to meet the desired confidence.
• Assures data collection ends when sufficient data
  exist.

21
Special TopicSoftware testing

• In an earlier lecture I presented the results of
  a study showing the economic impact of poor or
  limited software testing.
• Ive also made the case that earlier problem
  finding and fixing is exponentially cheaper.
• Here I want to introduce several software testing
  concepts.

22
Percent of Specification Requirements Involving
Software
F-22


B-2
F-16
F-15
F-111
A-7
F-4
Ref Lockheed - Martin Corp.
23
Special TopicSoftware testing

• Human testing is defined as informal,
  noncomputer-based methods of evaluating
  architectures, designs and interfaces. It can
  consist of
• Inspections The programmer explains his/her
  work to a small group of peers with discussion
  and direct feedback on errors, inconsistencies
  and omissions.
• Walk-through A group of peers develop test
  cases to evaluate work to date and give direct
  feedback to the programmer.
• Desk Checking A self evaluation is made by
  the programmer of his/her own work. There is a
  low probability of identifying his/her errors of
  logic or coding.

24
Special TopicSoftware testing
Peer Ratings Mutually supportive, anonymous
reviews are performed by groups of peers with
collaborative evaluations and feedback. Design
Reviews Preliminary design reviews (PDRs) and
critical design reviews (CDRs) provide milestones
in the development efforts that review
development and evaluations to date.
25
Special TopicSoftware testing

• Computerized software testing
• Black box
• Functional testing of a software unit without
  knowledge of how the internal structure or logic
  will process the input to obtain the specified
  output. Within-boundary and out-of-boundary
  stimulants test the softwares ability to handle
  abnormal events. Most likely cases are tested to
  provide a reasonable assurance that the software
  will demonstrate specified performance. Even the
  simplest software designs rapidly exceed capacity
  to test all alternatives.

26
Special TopicSoftware testing

• Computerized software testing
• White box
• Structural testing of the internal logic and
  software structure provides an opportunity for
  more extensive identification and testing of
  critical paths. The process and objectives are
  otherwise very similar to black box testing.

27
Special TopicSoftware testing

• IVV
• Independent verification and validation are
  risk-reducing techniques that are applied to
  major software development efforts. The primary
  purpose of IVV is to ensure that software meets
  requirements and is reliable and maintainable.
  The IVV is effective only if implemented early
  in the software development schedule.
  Requirements analysis and risk assessment are the
  most critical activities performed by IVV
  organizations.

28
Special TopicSoftware testing

• IVV
• The desire to boast of a high SEI rating in
  proposals and to generally improve the
  corporations efficiency has moved many companies
  to significantly improved software development
  capabilities.
• This in turn has reduced the perceived need by
  customers to require and pay for IVV.