AAA Automated Testing For AAA Games - PowerPoint PPT Presentation

Loading...

PPT – AAA Automated Testing For AAA Games PowerPoint presentation | free to view - id: 1ba223-ZDc1Z



Loading


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation
Title:

AAA Automated Testing For AAA Games

Description:

Clearly, this is not sustainable in the long term. Development ... 30% of bug fixes create a new bug caused by side effects. Excuses for not Testing (contd. ... – PowerPoint PPT presentation

Number of Views:62
Avg rating:3.0/5.0
Slides: 38
Provided by: Jes253
Category:

less

Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: AAA Automated Testing For AAA Games


1
AAA Automated TestingFor AAA Games
  • Francesco CarucciTechnical Director

2
(No Transcript)
3
How much does a game cost?
  • Development costs spiraling up
  • EA Exec Cost of games is crazy!
  • Some examples
  • Killzone 2 gt 40 M
  • Halo 3 gt 30 M
  • GTA 4 gt 100 M (rumored)
  • Clearly, this is not sustainable in the long term

4
Development cost partitioning
  • To keep it simple
  • Producing assets
  • Writing code (game code, technology, tools)
  • Debugging and maintaining code
  • Code dictates cost of producing assets
  • Better tools means less time to create assets
  • More efficient code means less time spent
    optimizing assets

5
Solutions?
  • Better production frameworks
  • SCRUM or some variation of it
  • More experienced teams
  • Higher senior/junior ratio actually costs less
  • Better tools (3rd party engines, off the shelf
    packages)
  • Out sourcing (yay for assets, nay for code)
  • Higher quality code

6
Code is not the King
  • But its a first class citizen and we want to
    keep its cost down
  • Higher code quality reduces Cost Of Ownership
  • Less bugs
  • Less over engineering
  • Higher code quality means code easier to change
    and iterate upon
  • Higher code quality means more time to be spent
    on polishing the game
  • Higher code quality means less money spent to
    make a game!
  • And a bigger chance to ship on time and hit that
    market sweet spot

7
Change Prevention Process
  • Many teams follow a change management process
    whose main goal is to prevent feature creep
  • Its effectively a Change Prevention Process
  • But you cant design fun up front
  • Games must be developed iteratively until fun
    is found
  • So?
  • Embrace Change ?

8
Cost of Change
  • Game Design requirements change
  • The aim system is good, but I would like it more
    peachy, you know a bit more of this, but slightly
    less of that, more rocking, less oompfh. Can you
    change it?
  • Traditionally Cost Of Change rises sharply over
    time
  • No, I cant change it, It would break X and Y
    and probably Z too, I dont really know
  • We want to keep Cost Of Change as flat as
    possible
  • Yes we can!

9
Enter Automated Testing
  • Automated Testing has been a broadly accepted
    tool for several decades in the Software Industry
    aimed at reducing cost of developing working code
  • In fact (warning Boring Slides Ahead)

10
Assessing test-driven development at IBM
  • In a software development group of IBM Retail
    Store Solutions, we built a non-trivial software
    system based on a stable standard specification
    using a disciplined, rigorous unit testing and
    build approach based on the test- driven
    development (TDD) practice. Using this practice,
    we reduced our defect rate by about 50 percent
    compared to a similar system that was built using
    an ad-hoc unit testing approach. The project
    completed on time with minimal development
    productivity impact. Additionally, the suite of
    automated unit test cases created via TDD is a
    reusable and extendable asset that will continue
    to improve quality over the lifetime of the
    software system. The test suite will be the basis
    for quality checks and will serve as a quality
    contract between all members of the
    team.http//collaboration.csc.ncsu.edu/laurie/P
    apers/MAXIMILIEN_WILLIAMS.PDFhttp//portal.acm.or
    g/citation.cfm?id776892

11
An initial investigation of test driven
development in industry
  • Simposium on Applied Computing
  • Test Driven Development (TDD) is a software
    development practice in which unit test cases are
    incrementally written prior to code
    implementation. In our research, we ran a set of
    structured experiments with 24 professional pair
    programmers. One group developed code using TDD
    while the other a waterfall-like approach. Both
    groups developed a small Java program. We found
    that the TDD developers produced higher quality
    code, which passed 18 more functional black box
    test cases. However, TDD developer pairs took 16
    more time for development. A moderate correlation
    between time spent and the resulting quality was
    established upon analysis. It is conjectured that
    the resulting high quality of code written using
    the TDD practice may be due to the granularity of
    TDD, which may encourage more frequent and
    tighter verification and validation. Lastly, the
    programmers which followed a waterfall-like
    process often did not write the required
    automated test cases after completing their code,
    which might be indicative of the tendency among
    practitioners toward inadequate testing. This
    observation supports that TDD has the potential
    of increasing the level of testing in the
    industry as testing as an integral part of code
    developmenthttp//portal.acm.org/citation.cfm?i
    d952753dlGUIDEcollGUIDECFID46811546CFTOKEN
    92849791

12
Google
  • Standard Google practices include unit tests and
    functional tests, continuous builds, last known
    good build, release branches with authorized bug
    fix check-ins, focused teams for release
    engineering, production engineering and QA, bug
    tracking and logging of production
    runs.http//agile2007.agilealliance.org/index.p
    hp3Fpagesub252Fid713.html

13
Professionalism and Test-Driven Development
  • Test-driven development is a discipline that
    helps professional software developers ship
    clean, flexible code that works, on time. In this
    article, the author discusses how test-driven
    development can help software developers achieve
    a higher degree of professionalismhttp//ieeexp
    lore.ieee.org/Xplore/login.jsp?urlhttp3A2F2Fie
    eexplore.ieee.org2Fiel52F522F41630082F04163026
    .pdfauthDecision-203

14
Unit Testing in Games
  • In our experience, the introduction of automated
    tests and continuous integration makes
    development teams more efficient and results in
    more reliable, and often quite simply, better
    software. Additionally, it reduces the pressure
    and workload on development teams by reducing the
    effort for manual testing, and allows bugs to be
    found earlier in the development process.
    Certainly, automated tests alone won't make your
    game a hit. But almost as certainly, they will
    make life easier for developers, artists, project
    leaders, producers and even players.http//www.
    gamasutra.com/features/20050329/roken_pfv.htm

15
Unit Testing in Games (contd)
  • Test-driven development can be a very effective
    development technique. We have successfully
    applied it to game development in a variety of
    situations, and were convinced of the many
    benefits it has provided us. Right now, the idea
    of writing code without writing tests first feels
    quite alien to most of us, and we treat TDD like
    the scaffolding in building construction a
    necessary tool that will not be shipped to the
    customer but that helps tremendously during
    developmenthttp//gamesfromwithin.com/?p50

16
Unit Testing in Games (contd)
  • Wir haben in den letzten fünf Jahren
    ausschließlich positive Erfahrungen mit
    automatisierten Tests und Continuous Integration
    gesammelt. Die automatisierten Tests sichern
    Stabilität auf hohem Niveau, während Continuous
    Integration zusätzlich eine deutliche
    Arbeitserleichterung darstellt. Der Aufwand für
    die Einrichtung und Wartung der Verfahren ist
    überschaubar und hat sich schnell amortisiert.
    Wenn man die anfängliche Skepsis überwindet,
    möchte man unserer Erfahrung nach diese Art der
    automatisierten Qualitätssicherung sehr schnell
    nie mehr missen.http//www.igda.dimajix.net/fil
    eadmin/ressources/report-03/07_roeken_summary.pdf

17
Whats an Automated Unit Test
  • A snippet of code that exercise a Unit Under Test
    (method of a class), assert its correctness and
    is executed automatically every time the system
    is built
  • Running unit test is effectively a compilation
    step
  • Compiler checks if the code is syntactically
    correct
  • Unit tests check if the code is semantically
    correct

18
Test First vs Test Last
  • Writing automated tests is accepted to produce
    higher quality code at lower cost
  • More tests less time spent debugging
  • Should tests be written after production code?
    (Test-Last)
  • Should tests be written before production code?
    (Test-First)
  • This observation supports that TDD has the
    potential of increasing the level of testing in
    the industry as testing as an integral part of
    code development
  • Test Driven Development Test-First

19
Test-Driven Development
  • Write a failing test before the code is written
  • Write the simplest code that could possibly make
    the test pass
  • Refactor the code to eliminate all possible
    duplications and code smells
  • Red Green Refactor
  • Thats the coding rhythm

20
A Failing Test
  • First write a failing test
  • CRY_TEST (FireButtonPressAndRelease) StartFire(
    ) ASSERT_IS_TRUE(IsFiring())
  • The test is dictating the interface and the
    contract of the unit being written
  • When StartFire method is invoked, the weapon is
    firing
  • then implement the minimal code to make it pass
  • bool IsFiring() const return true

21
Another Failing Test
  • Write another failing test to dictate more
    behavior
  • CRY_TEST (FireButtonPressAndRelease)
    StartFire() StopFire() ASSERT_IS_FALSE(IsFi
    ring())
  • When StartFire method is invoked, and then
    StopFire method is invoked, the weapon is not
    firing.
  • The actual word explanation is redundant, the
    whole information needed is in the code itself in
    readable form!

22
Another Failing Test (contd.)
  • Minimal implementation
  • bool StartFiring()
  • m_isFiring truebool StopFiring() m_isF
    iring falsebool IsFiring() const
  • return m_isFiring

23
How does it work
  • Tests are small, simple, straightforward and test
    only one condition
  • Long and complicated tests must be avoided
  • Tests are self documenting
  • Tests describe the design of the code to be
    written
  • Contract
  • Invariants
  • Post-conditions
  • Calling conventions
  • A TDD cycle should take few minutes (often less
    than a minute)
  • 30 minutes to write a test means a bad test
  • Break bigger tests in smaller steps (help test
    coverage)
  • Begin with baby steps, increase step size when
    becoming more comfortable

24
Why TDD?
  • Small and clearly defined steps lead to the final
    implementation
  • No over engineering write only the code that
    makes failing test pass
  • Minimal maintenance
  • Less code for the same needed functionality
    (including tests!)
  • Less time needed to change the code when
    requirements change
  • Less coupling between units
  • Less cost to maintain units
  • Tests are a by product of design and code
    construction
  • They keep contracts tested automatically through
    the life of the project
  • Code is easier to modify, broken contracts easier
    to discover
  • Defects and design inconsistencies are found
    early on
  • It costs less to fix fewer bugs!

25
When TDD is not appropriate
  • Exploratory coding, throw away prototype code to
    quickly explore different solutions for a problem
  • But the proper solution must be implemented
    test-driven to reduce cost
  • Code wrapping around existing libraries
  • GUI layer
  • Very low level code dealing with the hardware

26
Excuses for not Testing
27
Excuses for not Testing (contd.)
  • writing tests takes time and this time is
    stolen from writing code
  • People forget about the time spent debugging and
    maintaining code which is much larger than the
    time spent writing it
  • Tests written test-first are part of the design
    process, that has to be done anyway. TDD is just
    a formalized way of designing code. Overall it
    takes less time to produce working code
    test-driven
  • When the code base is covered, modifying code is
    safer and takes less time cause of regression
    provided by the automated tests
  • By writing tests first, fewer bugs are
    experienced by working with generally tested code
    (no huge amount of time spent debugging)

28
Excuses for not Testing (contd.)
  • tests cant cover everything
  • Some test coverage is still better than no test
    coverage at all
  • Legacy code shouldnt be tested unless its being
    refactored
  • Rome wasnt built in a day, but it was built
    eventually (and conquered the known world ?)

29
Excuses for not Testing (contd.)
  • but this code is temporary and I will change it
    later, I dont need to test it now
  • All code will be modified soon or later making
    code simpler to change is the main reason to
    write it test-driven reduce over engineering to
    the minimum, keep it simple
  • Requirements change modify the affected tests
    and production code. If unit tested, the code
    base is more decoupled and easier to modify with
    less side effects

30
Excuses for not Testing (contd.)
  • game development is different automated
    testing might not work
  • Game development as a whole is different
  • but writing code is the same if not simpler
  • Game code is mostly algorithmic, which is easier
    to test than GUI code, for example
  • An A algorithm is mostly the same, be it in a
    game or in a web crawler
  • Game code changes often more changes means more
    pressure for automated tests to protect against
    side effects

31
Excuses for not Testing (contd.)
  • games are about fun you cant capture fun with
    automated tests
  • But you can capture bugs, and less time spent
    debugging means more time looking for the fun
    part
  • Writing a test for a bug fix ensures that bug
    will not resurface later on
  • 30 of bug fixes create a new bug caused by side
    effects

32
Excuses for not Testing (contd.)
  • this is AAA Game, its too risky to do
    automated testing on it
  • AAA game means highly polished, highly
    technologic, great game that delivers great fun
  • In order to achieve AAA, the most efficient
    production tools are needed, Automated Testing is
    an efficient tool
  • Its too risky to NOT do automated testing on a
    AAA game
  • Be among the first to gain the competitive
    advantage and improve the Industry

33
A Slice of a bigger Pie
  • Automated tests (even test-driven) dont solve
    all problems
  • Chatting up a girl will still be a problem for a
    programmer ?
  • Automated tests must be part of a wider agile
    framework including
  • Continuous integration
  • Team code ownership
  • Daily deliverables
  • Simple code that works
  • Functional and Integration testing
  • These practices work together and strengthen each
    other with the goal of producing more
    functionalities in less time.

34
Whats next?
  • More automated tests beyond Unit testing
  • Functional testing
  • Test more units functioning together at higher
    level
  • Requires more time to build a Functional Testing
    Framework
  • Can capture bugs generated by the interaction of
    different units
  • Smoke testing
  • Load different levels after every check-in
    looking for crashes
  • Run scripted sequences looking for crashes
  • Generate random inputs looking for crashes

35
Whats next? (contd.)
  • Asset testing
  • Load different assets from the engine to check
    for crashes and correctness
  • Animations, Meshes, Textures
  • Example a Mesh must be smaller than 5000
    polygons if its a character, 20000 polygons if
    its a building and must have a physics mesh
    smaller than 100 polygons associated
  • It requires metadata
  • Run tests after every asset check-in by content
    creators!
  • Validating assets before they get into the build
    can save an enormous amount of debugging time

36
Adopting Automated Testing
  • It doesnt matter how much you read about it,
    its like riding a bike you need to try it to
    learn it and understand it
  • It needs strong leadership, clear vision, courage
    and a company willing to invest in Agile
    Practices
  • Lots of money to be saved and huge competitive
    advantage are at stake here
  • Dont take decisions without understanding the
    issues
  • Automated Testing often goes against perceived
    common practices in the Game Industry
  • It needs a lot of mentoring and training (writing
    good tests is difficult)
  • It needs strong commitment by management
  • The whole team must be bought into it
  • The rogue primadonnas unwilling to improve their
    skills must be removed from the team

37
Questions?
  • Shoot, I can take it ?
  • Email
  • francesco_at_crytek.com
  • francesco_at_carucci.org
About PowerShow.com