First time right

1
Exploratory Testing

• First time right !

By Derk-Jan de Grood Date April
2007 Location VVSS 2007 Eindhoven
2
Objectives for this presentation

• Provide insight in added value and pit-falls
• Give practical tips
• Reduce the hesitation on applying ET

3
Content

• Introduction.
• What is ET?
• Application of ET within KPN-Beta.
• Pitfalls and lessons learned.
• Evaluation of ET.
• Conclusion.
• Further reading.

4
Introduction- In the canteen
I am on a test-job, There are no
specifications, An external party builds the
application, And, we dont have much time !
Do you know Exploratory testing ?
5
Exploratory testing
6
What is Exploratory Testing
The no 1. excuse for not having to prepare our
test design in full detail. We do exploratory
testing !
An approach for unscripted testing based upon
skills and experience of the tester. ET is a risk
based technique using a formal procedure, test
charters and heuristics.
Exploratory testing is simultaneous learning,
test design, and test execution. James Bach
7
Traditional techniques
Error that remains undiscovered
Error found with used techniques
Error in the s/w Coverage of the testdesign
8
Philosophy of ET
1. Points of Interest (POI)
2. First tests executed
3. Plan next step based upon test results
4. Define new POI
5. Cont. with next POI
6. Finished?
Error in the s/w Coverage of the testdesign
9
Building our testdesign
BVA
Syntax
PCT
EP
Exploratory
the puzzle changes the puzzling. James Bach
10
Process Experience
11
ET Process

• Define test team
• Organize kick-off
• Define the POI (test charters)
• Assign charters
• Execute tests
• Debriefing
• Plan re- regression tests

12
Define team Kick-off

• Team consisted of
• 2 junior tester
• 2 system experts
• Moderator

Using junior testers worked well. They have
proven to be eager, flexible and creative.
13
Define Charter

• Charter ID
• Priority
• Estimated time
• Aimed result
• Why should we test this?
• Expected problems
• Not included in this Charter
• Conclusion

14
Charter- the conclusion
15
The iterative part of the process
16
The content of the test charters

• How do you divide all functionality and risks
  that need to be tested over the test charters.
  Our experience with scenario and function based
  testing?

17
Agility in test process

• When unacquainted with ET, expect the process to
  change over time.
• Debriefings were used to discus process.
• Make decisions.

Basic rule in agility Decide on what you know at
the time. Dont loose momentum by predicting
what you might know tomorrow.
18
Observations Decisions
19
De-briefing sessions

• Status overview of the carters
• Charters in print, or beamer
• Stakeholders group decisions

20
The ideal test-session length

• Session length Workshops ET held at Collis
  learned that On 4 hour session take 30
  minutes preparation3 hour test execution
  finding registration15 minutes evaluation
• One debriefing session a day(but start with 2
  sessions)

21
Dedication

• Pair testing together
• Synchronize working time
• Who does the operational tasks ?

Pair testing
Tester 1
Tester 2
New charters Retest Operational tasks
22
Test execution

• Inch deep / mile wide.
• Always register the findings during execution.
• Show-stopper leads to negative conclusion.
• Re-testing- only the showstopper.
• No regression testing.

23
Adding and adapting test charters

• Original based on functional breakdown
• Added charters for
• Scenarios
• Changes affecting other functions
• Multi user/Authorisation/User friendliness
• Error guessing (last day)

24
Testlog

• Tests are logged in the charter
• Used for regression testing

25
Time-boxing

• Assign more work than fits in the time-box.
• When charter not finished- explain during
  debriefing (inch deep-mile wide).
• Prioritize the remaining work and charters.
• At the end All agreed upon the priorities.

26
Planning

• What overhead activities do we need to take into
  account?
• Update the charter overview
• Creating new charters
• The debriefing sessions
• Finding meeting
• Release meeting
• Re-testing
• Executing new charters (100-3050120).

27
Evaluation
The ET session gave us clear understanding of the
quality of the system. This was achieved in a
very short period. The fun about ET is that its
fundamentals are easily understood. Jaap
Azier (KPN)
In order to use ET effectively we need to take
the lessons learned into account. In special the
logging and scenario testing. Still I am glad
we did ET. It enabled other people to get insight
in the quality of the system. Carin Smits
(KPN)
The project went well, great team working.
Together we worked towards the best working
method. In the end, we certainly have found it.
This resulted in clear and traceable test
results. Jasper Overgaauw Testexpert (Collis)
Exploratory testing is testing on the edge. ET
means taking the most out of people, this implies
your dealing with people issues. It is exciting
to find the edge of we have tested all the
essential. Hugo Achthoven Implementation
Manager (KPN)

• Close collaboration between users and test expert
  enabled provide a judgment on the quality of the
  system, in a only short term.
• Matthijs Jorissen
• Testexpert (Sogeti)

28
Wrap-up
29
Further reading
Chapter on ET is basis for this
presentationISBN9789012118835 Available early
2007
Chapter on ET by James BachISBN 9072194659
http//www.satisfice.com/ http//nl.wikipedia.org
/wiki/Exploratory_testing http//www.improveqs.n
l/ And many others
30
Questions?
Derk-Jan de Grood grood_at_Collis.nl Collis De
Heijderweg 1 2314 XZ Leiden The Netherlands
Acknowledgements Jaap Azier KPN Telecom Jasper
Overgouw - Collis