Test Abstractions

1
Test Abstractions
Example A Blackjack Machine
Check if the computer obeys the dealer rules when
it has 17 or more points
Intent
Player Immediately stands. Dealer should stand
when dscore is gt 17
Nat. Lang. Spec.
pStand lt 0 Always _at_ (dHit) if (dscore gt 17)
display(error)
HW Behavioral

• Tests can be described at different abstraction
  levels
• Tests can be defined top-down or bottom-up

2
Black Box Testing

• Testing without knowledge of the internals of the
  design.
• Only know system inputs/outputs
• Testing the functionality without considering
  implementation
• Inherently top-down

Black Box Testing Examples

1. Test a multiplier by supplying random numbers to
   multiply
2. Test an anti-lock braking system by hitting the
   brakes at different speeds
3. Test a traffic light controller by simulating
   pre-recorded traffic patterns

3
Black Box Testing Issues

• Advantages
• Tests can be generated early in the design
  process
• Tests can often be reused after design change
• Disadvantages
• Less information is available for test generation
• More manual process because no simulatable info
  exists

Example Random test generation
4
Specification-Based Test

• A block box testing approach
• Natural language specification is used to
  generate test sequence
• Goal is to generate tests which verify all
  aspects of the specification

Extract a feature
Create nat. lang tests
Create executable tests
Until all features have tests

• Clear relationship between features and tests
• Good to accommodate specification change

5
Extracting Features from a Specification

• Purely manual process
• Specs are long and vague

Part of a Blackjack Specification
The dealer must continue to take cards ("hit")
until his total is 17 or greater. An Ace in the
dealer's hand is always counted as 11 if possible
without the dealer going over 21.

• Each sentence could be a feature for testing

6
A Feature from Blackjack
A blackjack, or natural, is a total of 21 in your
first two cards. If you split a pair of Aces for
example, and then draw a ten-valued card on one
of the Aces, this is not a blackjack, but rather
a total of 21. The distinction is important,
because a winning blackjack pays the player odds
of 3 to 2.

• This detail is exactly something that a designer
  might miss
• Maybe the designer doesnt appreciate the
  difference (3 to 2 odds)

7
Another Feature from Blackjack
If the dealer turns an up-card of an Ace, he will
offer "Insurance" to the players. Insurance bets
can be made by betting up to half your original
bet amount. The dealer will check to see if he
has a 10-value card underneath his Ace, and if he
does have Blackjack, your winning Insurance bet
will be paid at odds of 21.

• Do you know what an up-card is?
• Is it clear that a player making an insurance bet
  ends up breaking even if the dealer has
  blackjack?

• Gamblers understand this, designers may not.
• Specifications often assume the background of the
  reader

8
Natural Language Test Descriptions

• For each feature, describe tests to validate each
  feature

• Requirements of a Test Description
• Describe sequence at controllable points
• Variables/signals which can be controlled during
  testing (inputs or internal)
• Describe sequence at observable points
• Variables/signals where results are known
  (outputs or internal)
• Describe timing if relevant to the application

9
Controllable and Observable Points

• Need to select variable/signals to apply test
  data and to observe test results
• Need minimal structural detail (I/O, etc.)
• Controllable points - Inputs are the obvious
  choice
• Other points are chosen to reduce test time or
  control randomness
• Setting an internal 8 bit counter value
• Bypassing a random process

• Observable points - Output are the obvious points
• Other points are chosen to reduce test time and
  localize errors
• Observe internal regs in a microprocessor

10
Sample Test Description
The dealer must continue to take cards ("hit")
until his total is 17 or greater.

• The players choices should not matter
• Will any test validate this feature? No.
• - dealer has 16 and draws a 6
• Need to control the random deal

Possible test sequence Player stands, dealer
receives 10 and 7.
11
Sample Test Description
A blackjack, or natural, is a total of 21 in your
first two cards. If you split a pair of Aces for
example, and then draw a ten-valued card on one
of the Aces, this is not a blackjack, but rather
a total of 21. The distinction is important,
because a winning blackjack pays the player odds
of 3 to 2.

• Test this by causing the player to make a bet,
  draw a blackjack, and check the winnings

Possible test sequence Player bets X dollars.
Player hits and draws an ace. Player hits and
draws a 10. Verify 3 to 2 payoff.
12
Create Executable Tests

• Write the verilog test bench to implement each
  test
• Need to know all controllable and observable
  points in detail

Test sequence Player stands, dealer receives 10
and 7.
module BlackJack (player_choice, dealt_card,
dealer_choice, ) player_choice 1 // stand If
(dealer_choice ! 0) display (error) 1
dealt_card 10 If (dealer_choice ! 0)
display(error) 1 dealt_card 7 If
(dealer_choice ! 1) display (error)