Evaluating Testing Methods by Delivered Reliability

1
Evaluating Testing Methods by Delivered
Reliability
Section 3.4

• Frankl, Hamlet, Littlewood, Strigini
• IEEE TOSE Aug98

2
3.2 SFR, w/o subdomains

• d StinF V(t)
• P(Q0) 1-(1-d)T
• P(Qq) (1-d)T
• E(Q) 0 P(Q0) q P(Qq)
• q(1-d)T

3
(No Transcript)
4
3.3 single failure region, with sub

• Debug with n subdomains
• E(Q) qP(1-di)T/n
• Operational
• E(Q) q(1-q)T

5
Assumptions multiple failure model
6
Multiple Failure,debug w/o sub

• E(Q) Sqi(1-di)T
• E(Q) Sqi(1-qi)T

7
Example 1 assume 100 tests in domain
8
Example 2 Debug w/o subdomains?
assume disjoint
9
Example 3 subdomain testing
10
Example 3 Solve empirically
11
Empirical Tool

• Abstract domain to integers 1-100
• Abstract failure sets into number ranges
• Run N (upto 5K) test tests
• Each test set may have subdomains or n random
  tests
• Determine frequency of each detect/notdetect
  combination
• Input q of each detect/notdetect combination
• Calculate E(Q)

12
Example Code
for(i1iltMAXi) answer 0 int j for
(j1jlt6j) test 1 rand()80 if
(test lt 10) answer answer 1 if (test gt
20 test lt 25 ) answer answer
2 error2 1 if ( test gt 30
test lt35 error2 0) answer answer
2 if (test gt 35 test lt 36)answer
answer 4 error4 1 if (test gt
36 test lt 37) answer answer 8 if
(testgt 30 test lt 32 error4 0) answer
answer 4 if (test gt 60 test lt 62
error4 0) answer answer 4 if (test gt
62 test lt 63) answer answer
16 foundanswer
13
For Tuesday, Sep 18

• Read rest of article
• Build tool
• Do experiments for subdomain testing with
  multiple faults.