AAA Automated Testing For AAA Games

1
AAA Automated TestingFor AAA Games

• Francesco CarucciTechnical Director

2
(No Transcript)
3
How much does a game cost?

• Development costs spiraling up
• EA Exec Cost of games is crazy!
• Some examples
• Killzone 2 gt 40 M
• Halo 3 gt 30 M
• GTA 4 gt 100 M (rumored)
• Clearly, this is not sustainable in the long term

4
Development cost partitioning

• To keep it simple
• Producing assets
• Writing code (game code, technology, tools)
• Debugging and maintaining code
• Code dictates cost of producing assets
• Better tools means less time to create assets
• More efficient code means less time spent
  optimizing assets

5
Solutions?

• Better production frameworks
• SCRUM or some variation of it
• More experienced teams
• Higher senior/junior ratio actually costs less
• Better tools (3rd party engines, off the shelf
  packages)
• Out sourcing (yay for assets, nay for code)
• Higher quality code

6
Code is not the King

• But its a first class citizen and we want to
  keep its cost down
• Higher code quality reduces Cost Of Ownership
• Less bugs
• Less over engineering
• Higher code quality means code easier to change
  and iterate upon
• Higher code quality means more time to be spent
  on polishing the game
• Higher code quality means less money spent to
  make a game!
• And a bigger chance to ship on time and hit that
  market sweet spot

7
Change Prevention Process

• Many teams follow a change management process
  whose main goal is to prevent feature creep
• Its effectively a Change Prevention Process
• But you cant design fun up front
• Games must be developed iteratively until fun
  is found
• So?
• Embrace Change ?

8
Cost of Change

• Game Design requirements change
• The aim system is good, but I would like it more
  peachy, you know a bit more of this, but slightly
  less of that, more rocking, less oompfh. Can you
  change it?
• Traditionally Cost Of Change rises sharply over
  time
• No, I cant change it, It would break X and Y
  and probably Z too, I dont really know
• We want to keep Cost Of Change as flat as
  possible
• Yes we can!

9
Enter Automated Testing

• Automated Testing has been a broadly accepted
  tool for several decades in the Software Industry
  aimed at reducing cost of developing working code
• In fact (warning Boring Slides Ahead)

10
Assessing test-driven development at IBM

• In a software development group of IBM Retail
  Store Solutions, we built a non-trivial software
  system based on a stable standard specification
  using a disciplined, rigorous unit testing and
  build approach based on the test- driven
  development (TDD) practice. Using this practice,
  we reduced our defect rate by about 50 percent
  compared to a similar system that was built using
  an ad-hoc unit testing approach. The project
  completed on time with minimal development
  productivity impact. Additionally, the suite of
  automated unit test cases created via TDD is a
  reusable and extendable asset that will continue
  to improve quality over the lifetime of the
  software system. The test suite will be the basis
  for quality checks and will serve as a quality
  contract between all members of the
  team.http//collaboration.csc.ncsu.edu/laurie/P
  apers/MAXIMILIEN_WILLIAMS.PDFhttp//portal.acm.or
  g/citation.cfm?id776892

11
An initial investigation of test driven
development in industry

• Simposium on Applied Computing
• Test Driven Development (TDD) is a software
  development practice in which unit test cases are
  incrementally written prior to code
  implementation. In our research, we ran a set of
  structured experiments with 24 professional pair
  programmers. One group developed code using TDD
  while the other a waterfall-like approach. Both
  groups developed a small Java program. We found
  that the TDD developers produced higher quality
  code, which passed 18 more functional black box
  test cases. However, TDD developer pairs took 16
  more time for development. A moderate correlation
  between time spent and the resulting quality was
  established upon analysis. It is conjectured that
  the resulting high quality of code written using
  the TDD practice may be due to the granularity of
  TDD, which may encourage more frequent and
  tighter verification and validation. Lastly, the
  programmers which followed a waterfall-like
  process often did not write the required
  automated test cases after completing their code,
  which might be indicative of the tendency among
  practitioners toward inadequate testing. This
  observation supports that TDD has the potential
  of increasing the level of testing in the
  industry as testing as an integral part of code
  developmenthttp//portal.acm.org/citation.cfm?i
  d952753dlGUIDEcollGUIDECFID46811546CFTOKEN
  92849791

12
Google

• Standard Google practices include unit tests and
  functional tests, continuous builds, last known
  good build, release branches with authorized bug
  fix check-ins, focused teams for release
  engineering, production engineering and QA, bug
  tracking and logging of production
  runs.http//agile2007.agilealliance.org/index.p
  hp3Fpagesub252Fid713.html

13
Professionalism and Test-Driven Development

• Test-driven development is a discipline that
  helps professional software developers ship
  clean, flexible code that works, on time. In this
  article, the author discusses how test-driven
  development can help software developers achieve
  a higher degree of professionalismhttp//ieeexp
  lore.ieee.org/Xplore/login.jsp?urlhttp3A2F2Fie
  eexplore.ieee.org2Fiel52F522F41630082F04163026
  .pdfauthDecision-203

14
Unit Testing in Games

• In our experience, the introduction of automated
  tests and continuous integration makes
  development teams more efficient and results in
  more reliable, and often quite simply, better
  software. Additionally, it reduces the pressure
  and workload on development teams by reducing the
  effort for manual testing, and allows bugs to be
  found earlier in the development process.
  Certainly, automated tests alone won't make your
  game a hit. But almost as certainly, they will
  make life easier for developers, artists, project
  leaders, producers and even players.http//www.
  gamasutra.com/features/20050329/roken_pfv.htm

15
Unit Testing in Games (contd)

• Test-driven development can be a very effective
  development technique. We have successfully
  applied it to game development in a variety of
  situations, and were convinced of the many
  benefits it has provided us. Right now, the idea
  of writing code without writing tests first feels
  quite alien to most of us, and we treat TDD like
  the scaffolding in building construction a
  necessary tool that will not be shipped to the
  customer but that helps tremendously during
  developmenthttp//gamesfromwithin.com/?p50

16
Unit Testing in Games (contd)

• Wir haben in den letzten fünf Jahren
  ausschließlich positive Erfahrungen mit
  automatisierten Tests und Continuous Integration
  gesammelt. Die automatisierten Tests sichern
  Stabilität auf hohem Niveau, während Continuous
  Integration zusätzlich eine deutliche
  Arbeitserleichterung darstellt. Der Aufwand für
  die Einrichtung und Wartung der Verfahren ist
  überschaubar und hat sich schnell amortisiert.
  Wenn man die anfängliche Skepsis überwindet,
  möchte man unserer Erfahrung nach diese Art der
  automatisierten Qualitätssicherung sehr schnell
  nie mehr missen.http//www.igda.dimajix.net/fil
  eadmin/ressources/report-03/07_roeken_summary.pdf

17
Whats an Automated Unit Test

• A snippet of code that exercise a Unit Under Test
  (method of a class), assert its correctness and
  is executed automatically every time the system
  is built
• Running unit test is effectively a compilation
  step
• Compiler checks if the code is syntactically
  correct
• Unit tests check if the code is semantically
  correct

18
Test First vs Test Last

• Writing automated tests is accepted to produce
  higher quality code at lower cost
• More tests less time spent debugging
• Should tests be written after production code?
  (Test-Last)
• Should tests be written before production code?
  (Test-First)
• This observation supports that TDD has the
  potential of increasing the level of testing in
  the industry as testing as an integral part of
  code development
• Test Driven Development Test-First

19
Test-Driven Development

• Write a failing test before the code is written
• Write the simplest code that could possibly make
  the test pass
• Refactor the code to eliminate all possible
  duplications and code smells
• Red Green Refactor
• Thats the coding rhythm

20
A Failing Test

• First write a failing test
• CRY_TEST (FireButtonPressAndRelease) StartFire(
  ) ASSERT_IS_TRUE(IsFiring())
• The test is dictating the interface and the
  contract of the unit being written
• When StartFire method is invoked, the weapon is
  firing
• then implement the minimal code to make it pass
• bool IsFiring() const return true

21
Another Failing Test

• Write another failing test to dictate more
  behavior
• CRY_TEST (FireButtonPressAndRelease)
  StartFire() StopFire() ASSERT_IS_FALSE(IsFi
  ring())
• When StartFire method is invoked, and then
  StopFire method is invoked, the weapon is not
  firing.
• The actual word explanation is redundant, the
  whole information needed is in the code itself in
  readable form!

22
Another Failing Test (contd.)

• Minimal implementation
• bool StartFiring()
• m_isFiring truebool StopFiring() m_isF
  iring falsebool IsFiring() const
• return m_isFiring

23
How does it work

• Tests are small, simple, straightforward and test
  only one condition
• Long and complicated tests must be avoided
• Tests are self documenting
• Tests describe the design of the code to be
  written
• Contract
• Invariants
• Post-conditions
• Calling conventions
• A TDD cycle should take few minutes (often less
  than a minute)
• 30 minutes to write a test means a bad test
• Break bigger tests in smaller steps (help test
  coverage)
• Begin with baby steps, increase step size when
  becoming more comfortable

24
Why TDD?

• Small and clearly defined steps lead to the final
  implementation
• No over engineering write only the code that
  makes failing test pass
• Minimal maintenance
• Less code for the same needed functionality
  (including tests!)
• Less time needed to change the code when
  requirements change
• Less coupling between units
• Less cost to maintain units
• Tests are a by product of design and code
  construction
• They keep contracts tested automatically through
  the life of the project
• Code is easier to modify, broken contracts easier
  to discover
• Defects and design inconsistencies are found
  early on
• It costs less to fix fewer bugs!

25
When TDD is not appropriate

• Exploratory coding, throw away prototype code to
  quickly explore different solutions for a problem
• But the proper solution must be implemented
  test-driven to reduce cost
• Code wrapping around existing libraries
• GUI layer
• Very low level code dealing with the hardware

26
Excuses for not Testing
27
Excuses for not Testing (contd.)

• writing tests takes time and this time is
  stolen from writing code
• People forget about the time spent debugging and
  maintaining code which is much larger than the
  time spent writing it
• Tests written test-first are part of the design
  process, that has to be done anyway. TDD is just
  a formalized way of designing code. Overall it
  takes less time to produce working code
  test-driven
• When the code base is covered, modifying code is
  safer and takes less time cause of regression
  provided by the automated tests
• By writing tests first, fewer bugs are
  experienced by working with generally tested code
  (no huge amount of time spent debugging)

28
Excuses for not Testing (contd.)

• tests cant cover everything
• Some test coverage is still better than no test
  coverage at all
• Legacy code shouldnt be tested unless its being
  refactored
• Rome wasnt built in a day, but it was built
  eventually (and conquered the known world ?)

29
Excuses for not Testing (contd.)

• but this code is temporary and I will change it
  later, I dont need to test it now
• All code will be modified soon or later making
  code simpler to change is the main reason to
  write it test-driven reduce over engineering to
  the minimum, keep it simple
• Requirements change modify the affected tests
  and production code. If unit tested, the code
  base is more decoupled and easier to modify with
  less side effects

30
Excuses for not Testing (contd.)

• game development is different automated
  testing might not work
• Game development as a whole is different
• but writing code is the same if not simpler
• Game code is mostly algorithmic, which is easier
  to test than GUI code, for example
• An A algorithm is mostly the same, be it in a
  game or in a web crawler
• Game code changes often more changes means more
  pressure for automated tests to protect against
  side effects

31
Excuses for not Testing (contd.)

• games are about fun you cant capture fun with
  automated tests
• But you can capture bugs, and less time spent
  debugging means more time looking for the fun
  part
• Writing a test for a bug fix ensures that bug
  will not resurface later on
• 30 of bug fixes create a new bug caused by side
  effects

32
Excuses for not Testing (contd.)

• this is AAA Game, its too risky to do
  automated testing on it
• AAA game means highly polished, highly
  technologic, great game that delivers great fun
• In order to achieve AAA, the most efficient
  production tools are needed, Automated Testing is
  an efficient tool
• Its too risky to NOT do automated testing on a
  AAA game
• Be among the first to gain the competitive
  advantage and improve the Industry

33
A Slice of a bigger Pie

• Automated tests (even test-driven) dont solve
  all problems
• Chatting up a girl will still be a problem for a
  programmer ?
• Automated tests must be part of a wider agile
  framework including
• Continuous integration
• Team code ownership
• Daily deliverables
• Simple code that works
• Functional and Integration testing
• These practices work together and strengthen each
  other with the goal of producing more
  functionalities in less time.

34
Whats next?

• More automated tests beyond Unit testing
• Functional testing
• Test more units functioning together at higher
  level
• Requires more time to build a Functional Testing
  Framework
• Can capture bugs generated by the interaction of
  different units
• Smoke testing
• Load different levels after every check-in
  looking for crashes
• Run scripted sequences looking for crashes
• Generate random inputs looking for crashes

35
Whats next? (contd.)

• Asset testing
• Load different assets from the engine to check
  for crashes and correctness
• Animations, Meshes, Textures
• Example a Mesh must be smaller than 5000
  polygons if its a character, 20000 polygons if
  its a building and must have a physics mesh
  smaller than 100 polygons associated
• It requires metadata
• Run tests after every asset check-in by content
  creators!
• Validating assets before they get into the build
  can save an enormous amount of debugging time

36
Adopting Automated Testing

• It doesnt matter how much you read about it,
  its like riding a bike you need to try it to
  learn it and understand it
• It needs strong leadership, clear vision, courage
  and a company willing to invest in Agile
  Practices
• Lots of money to be saved and huge competitive
  advantage are at stake here
• Dont take decisions without understanding the
  issues
• Automated Testing often goes against perceived
  common practices in the Game Industry
• It needs a lot of mentoring and training (writing
  good tests is difficult)
• It needs strong commitment by management
• The whole team must be bought into it
• The rogue primadonnas unwilling to improve their
  skills must be removed from the team

37
Questions?

• Shoot, I can take it ?
• Email
• francesco_at_crytek.com
• francesco_at_carucci.org