Issues and Implementations in Multiplayer Game Development - PowerPoint PPT Presentation


PPT – Issues and Implementations in Multiplayer Game Development PowerPoint presentation | free to download - id: 2674a-MjAxM


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation

Issues and Implementations in Multiplayer Game Development


Usually, cheats are enabled only in the single-player mode of the game (if it ... On some PC applications, even though there may be a server running on the same ... – PowerPoint PPT presentation

Number of Views:191
Avg rating:3.0/5.0
Slides: 118
Provided by: Lar9156
Learn more at:


Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: Issues and Implementations in Multiplayer Game Development

Issues and Implementations in Multiplayer Game
  • Online game development is difficult and riddled
    with expensive risks. Whether you are adding
    multiplayer support to a single-player game or
    fielding a massively multiplayer persistent
    virtual world, the additional complexity of
    connecting your players over a network adds
    substantial development time and testing costs.
    Get it right, and online can bring innovative
    game play and a sense of community to your player
    base. Get it wrong, and going online brings you a
    Pandoras box of missed deadlines, frustrated
    developers, and unhappy customers

Why Online?
Tutorial Takeaway Messages
  • Building online games with single-player game
    techniques is painful
  • Early changes to your development process
    system architecture to accommodate online play
    will greatly ease the pain
  • Lessons Learned from our background will
    provide your project with useful Rules of Thumb

A wise man learns from his own mistakes, a wiser
man learns from someone elses
Network Terminology
  • A measure of the width or capacity of a
    communications channel. Greater bandwidth allows
    communication of more information in a given
    period of time.
  • Important Bandwidth refers only to the amount of
    data a channel can handle. As an analogy, a dump
    truck has a higher bandwidth than a sports car.
    This does not mean that the dump truck is faster,
    just more efficient at moving large amounts of

Source Ubisoft Online Terminology Guide, used
with permission
  • An exception to a games normal rule set that is
    included by the games developers. Usually,
    cheats are enabled only in the single-player mode
    of the game (if it has one) or were included for
    use (and forgotten about) by the developers to
    aid with testing and/or the media.

  • This is any networking architecture in which one
    machine (the server) is responsible for making
    final determinations on what occurs in the game.
    This can range from game designs in which the
    client merely displays information to the user
    and returns user inputs to the server, to game
    designs in which all machines are running nearly
    complete versions of the game, and the server
    only serves to centralize communications and
    adjudicate any disagreements.
  • Important This is an architectural distinction.
    It is entirely possible that the server is a
    consumer operated machine.

Source Ubisoft Online Terminology Guide, used
with permission
  • A game in which all of the machines involved are
    run by the consumer.

Source Ubisoft Online Terminology Guide, used
with permission
Dedicated Server
  • A client-server game in which there is a machine
    which is serving as a server without having any
    local player. This may be hosted by a consumer,
    or by a game service.

Source Ubisoft Online Terminology Guide, used
with permission
  • A client-server game design in which the client
    is largely a dumb input/output device.

Source Ubisoft Online Terminology Guide, used
with permission
  • An unexpected or unintended game result
    generated by the actions of the player that
    modifies the course or outcome the game in a way
    not intended by the games designers. Usually
    the result of players discovering a bug in the

  • A modification made to a game or its data by an
    end user. This can take the form of modifying
    either code or data, in memory while the game is
    running, and/or to the games files on disk. The
    game itself or a service that it uses (such as a
    driver), or the data it shares with other
    computers, or hardware can be modified.

Hybrid Game
  • Mix between Client-Server and Peer-to-Peer,
    where a server determines the outcome for some
    events, while each players computer determine
    the outcome of some events (usually local to that

  • A security product that employs a combination of
    hardware and software to prevent unauthorized
    users or traffic from the Internet from gaining
    access to a private local area network. Many
    problems in online gaming arise because firewalls
    block communication between games.

Source Ubisoft Online Terminology Guide, used
with permission
Integrated Server
  • A client-server game in which the machine which
    has a user playing the game is also serving as
    the server. On some PC applications, even though
    there may be a server running on the same machine
    as a player, they are separate applications.

Source Ubisoft Online Terminology Guide, used
with permission
IP Address
  • This is the identifying number for any machine on
    the Internet or for any LAN using the IP
    protocol. The original, and most widely used
    form, is IPv4. Under IPv4, each address is made
    up of four octets. If expressed as text, it will
    take the form of "" (sometimes
    called a dotted IP string). The broader Internet
    is slowly switching over to IPv6 to increase the
    number of available addresses. As the name
    implies, IPv6 addresses are made up of six

Source Ubisoft Online Terminology Guide, used
with permission
IP Address
  • Additionally, under IPv4, the following rules
    apply to network addresses The local machine is
    always reachable by the "loopback" address of, and the following ranges of addresses
    are reserved for use on local LANs
  • -
  • -
  • -

Source Ubisoft Online Terminology Guide, used
with permission
  • A slowing of the execution of a game. Lag can be
    caused by two main factors network issues or
    computer performance.
  • Network lag cause an online game to miss
    information bits necessary for the execution of
    it. For instance, the last message indicating the
    position of another player can be lost or is
    late, but the game has to display something on
    the screen. Depending on how this situation has
    been taken into account by the developers, the
    display of the other player will freeze and the
    computer will wait for the next message to arrive
    before continuing the execution of the game.   

Source Ubisoft Online Terminology Guide, used
with permission
  • Performance lag is caused by the overload of the
    main CPU (too much information to process at a
    time) or the graphical CPU (too many polygons or
    lightning to process).

Source Ubisoft Online Terminology Guide, used
with permission
  • A measure of how long it takes a single bit to
    propagate from one end of a link or channel to
    the other. Latency is measured strictly in terms
    of time.

Source Ubisoft Online Terminology Guide, used
with permission
MAC Address
  • Media Access Control Address, a hardware address
    that has been embedded into the network interface
    card by the manufacturer to uniquely identify
    each node or point of connection of a network.

Source Ubisoft Online Terminology Guide, used
with permission
NAT (Network Address Translation)
  • Enables a local area network to use one set of IP
    addresses for internal traffic and a second set
    of IP addresses for external traffic.

Source Ubisoft Online Terminology Guide, used
with permission
  • An 8 bit value. Because a byte is the smallest
    addressable amount of memory on the machine (6
    bit, 7 bit, 9 bit, and 16 bit machines have all
    been on networks, along with the now standard 8
    bit bytes), network engineers like to use octet
    to specify that we really mean 8 bits.

Source Ubisoft Online Terminology Guide, used
with permission
  • In this networking architecture, all games are
    running full versions of the game engine.
    Although one machine may be used to host the
    setup of the game, once the game is running, all
    machines are effectively equal. Disputes may be
    resolved algorithmically (i.e. claims about your
    own damage are always valid) which lends itself
    to cheating, or via a voting mechanism.
  • Important This is an architecture. It does not
    mean any game where all the machines involved
    are consumer owned.

Source Ubisoft Online Terminology Guide, used
with permission
  • A client-server game design in which the clients
    are running relatively complete versions of the
    simulation, and the server is validating claims,
    rather than determining all results.

Source Ubisoft Online Terminology Guide, used
with permission
Service Hosted
  • A game in which one or more of the machines
    involved is provided by a game service (often the
    games publisher).

Source Ubisoft Online Terminology Guide, used
with permission
  • Transmission Control Protocol. TCP is one of the
    main protocols in TCP/IP networks. Whereas the IP
    protocol deals only with packets, TCP enables two
    hosts to establish a connection and exchange
    streams of data. TCP guarantees delivery of data
    and also guarantees that packets will be
    delivered in the same order in which they were

Source Ubisoft Online Terminology Guide, used
with permission
  • A connectionless protocol that, like TCP, runs on
    top of IP networks. Unlike TCP/IP, UDP/IP
    provides no error recovery services. UDP/IP
    packets are CRC checked, but if there is a CRC
    failure or part of a UDP/IP packet fails to
    arrive, the packet is lost entirely.
    Additionally, where TCP/IP is stream based,
    UDP/IP is datagram based, with all the data in a
    UDP/IP send arriving at once, if it arrives at
    all. It can be used for broadcasting messages
    over a local network to find games, and is used
    by many games as the primary game data channel.

Source Ubisoft Online Terminology Guide, used
with permission
What is a WAN?
  • Wide Area Network (WAN) is a network of
  • Allows different networks to talk to each other
  • Size and distance doesnt matter

LAN Environment
  • Local Area Network (LAN) shares common address
  • Actually independent of media usually Ethernet
  • Also independent of protocol usually TCP/IP
  • Can broadcast

WAN Environment
  • Multiple networks
  • Can have overlapping address spaces
  • Very media dependent
  • Uses edge device to hide the details
  • Different types based on connection details

WAN Connection Types
  • Point-to-point a single leased line between two
  • Circuit Switching like a phone call, connection
    made on demand
  • Packet switched Virtual circuits are created

Point to Point
  • A leased line from the provider
  • Always on
  • Fixed bandwidth
  • Works like a pipe, whatever goes in at one end
    comes out the other
  • Can set up an extended LAN environment
  • More expensive

Circuit Switching
  • Similar to a leased line in that there is a
    physical connection between the two sites
  • Gets set up and torn down so its only on when
  • Similar to making a phone call
  • ISDN is an example

Packet Switched
  • Uses provider infrastructure
  • No real connection between sites, uses a virtual
  • Routes data packets to final destination by
    whatever path is available
  • Two types of circuits Switched Virtual Circuit
    (SVC) and Permanent Virtual Circuit (PVC)
  • Examples are ATM, Frame Relay, X.25, SMDS

Virtual Circuits
  • PVCs are always on permanently established
  • SVCs are brought up and down
  • Call Setup
  • Idle
  • In use
  • Tear down

Quality of Service
  • Quality of Service (QOS) requires a queuing
  • Differs between media used
  • Prioritizes data according to packet information
  • Line purchased may or may not support - Make
    sure you buy the right thing!

Edge Device
  • Any device that hides the complexity of what you
    are talking to from your LAN
  • Does the protocol disassembly and reassembly
  • Handles signaling
  • Handles media conversion
  • Router is an example

  • Asynchronous Transfer Mode
  • Used by Telcos to handle voice
  • Breaks everything up into small, fixed size
  • Very common, cheap
  • T1s, T3s, OC3s etc.
  • Need to test for equipment compatibility!

Typical ATM Usage
  • Local Exchange Carrier (LEC) provides physical
  • Line is located in the carriers Point of
    Presence (POP) on your facility
  • You connect your router to your internal Ethernet
  • LEC connects line to your carrier

Typical ATM Usage cont.
  • Your carrier (Sprint, ATT, Global Crossing, etc.)
    has a small, fixed number of hops to get you onto
    main trunk
  • Main trunk will get your data to destination LEC
  • LEC to POP to Router to Destination LAN
  • Magic is in how it knows where to go!

Break until 1115
  • 1115am-1230pm
  • Moving from Single Player to Multiplayer

GAMESUse Case Steal Ball Being Dribbled By
Another Player
Remote machine always has an approximation of
ball position
Issue Kicking a Shared Soccer Ball
  • Your opponent has the _exact_ position of the
  • Your computer has only an _approximate_ position
    of the ball
  • When you (the player or the coder) make a
    decision, it may be on incorrect data
  • Level of distortion impacts quality of gameplay
  • Inaccuracies of data lead to serious coding

The more computers involved, the greater the
Shared Worldview Each Node Has A Potentially
Different Representation Of Every Value
Client A (Local Memory) Clear Views
Game Logic
Player Views
True Values
Damage Levels
Player Position
AI State
At any slice in time, fuzzy, loosely coupled
views produce an inconsistent shared worldview
Machine A, time t
Machine B , time t
Shared View _at_ (t) Is ALWAYS DIFFERENT
Machine C , time t
The loss of precision
  • Compression saves bandwidth
  • What to compress and how much?
  • Possibly compress then decompress before you use
    it in game to minimise divergence
  • Examples
  • Car position vibration
  • Car position aggressive driving

What is an atomic operation?
  • Each message becomes an atomic operation
  • Interframe and intraframe timing is not preserved
    across the network
  • Troublesome case Multiple frame actions such as
    shell transitions
  • Troublesome case Multi-step operations in which
    the interim cases are invalid at frame start

Event Ordering And Atomicity Across Transactions
Further Impact GamePlay And Coding Errors
Near-Endless (and illogical) Ordering Are Possible
Machine A
Machine B
A1,A2,B1,B2 A1, B1,B2,A2 B1,B2,A1,A2 A1,A2,B1,B2
A2,A1,B1,B2 B2,A2,B1,A1
Machine C
Indeterminate timing and sequencing of information
  • Player A sends a stream of packets describing
    game state
  • Timing
  • Player A picks up invulnerability pickup (10s)
  • Player A gets shot 9.9s later
  • If the second packet is delayed before arriving
    at B

Indeterminate timing and sequencing of information
  • Sequencing
  • Player A picks up rocket-launcher
  • Player A fires rocket
  • If these arrive in the wrong order at B

Atomicity Across Transactions(Sharing Internal
Client Process
Logical Thread (A)
Logical Thread (B)
  • Generate House Info Request (11,11)
  • Set Local State waiting for reply on house

Local House State (11,11)
  • Generate House Info Request (22,22)
  • Set Local State waiting for reply on house

Local House State (22,22)
3. EnterHouse (LocalHouseState)
Thread (A) acts off of incorrect data
TSO Solution Encapsulate State Logic in a
State Machine
Regulator A
Client Process
Regulator B
Logical Thread (A)
Logical Thread (B)
  • Generate House Info Request (11,11)
  • Set Local State waiting for reply on house

Local House State (11,11)
  • Generate House Info Request (22,22)
  • Set Local State waiting for reply on house

Local House State (22,22)
3. EnterHouse (LocalHouseState)
Thread (A) now acts off of correct data
Latency (and why it isn't constant)
  • The internet

Latency (and why it isn't constant)
  • Router buffers

Latency (and why it isn't constant)
  • Different routes

Latency (and why it isn't constant)
  • Packet loss

API options State Replication Versus Event Driven
Client A
Client B
State Changes
State Changes
Best Available representation of state
Best Available representation of state
Shared StateDB
Shared StateDB
Partial Updates
Intuitive Model Hides Network Optimizations
API can create new problems Hidden network costs
Recommendation Support Shared State Events
Client A
Client B
Immediate Events
State Changes
State Changes
Best Available representation of state
Best Available representation of state
Shared StateDB
Shared StateDB
Partial Updates
Shared State
Instrument for volume latency Optimize to
network budget
Use for stateless transactions, Immedidate actions
Shared WorldView Summary(Impact On Players
  • Player immersion jerky visuals, occasional
  • Fairplay who killed who?
  • Well, who shot first? How can you tell?
  • Was Player A actually located where Player B shot
  • Coding issues
  • Race conditions
  • Event ordering loss of precision can change
    future outcomes per machine
  • Bugs of this sort are hard to reproduce

Shared WorldView Summary (Mitigating Factors)
  • High degrees of precision are rarely required
  • Predictable motion patterns simplify hiding the
    latency (dead reckoning, predictive contracts, et
  • Rule of Thumb design game play to minimize the
    required accuracy (and volume) of shared data
  • Useful tactic handle all inter-process
    communication via state machines that handle each
    transaction from initiation to conclusion (TSOs
    Regulator template worked well)

Scale and Scaleability
  • Scale the N2 problem
  • Data Transmission Optimizations minimize the
    network cost of sharing the required amount of
    Shared State
  • Interest management, predictive contracts (dead
    reckoning), variable resolution, compression,
  • Application Scalability minimize the amount of
    required Shared State
  • References
  • Bandwidth these days, who cares?
  • Caveat see Scale!
  • Best bet metrics, metrics, metrics
  • Set budget per player, optimize to that point,
    then STOP
  • Danger pre-mature optimization
  • Danger game play changes that break optimizations

Why every and all are dangerous
  • SP all information is always available
  • MP shared information must be packaged,
    transmitted and unpackaged
  • Each step costs CPU bandwidth, and can happen
    10s to 100s of times per minute
  • May also cause additional overhead (e.g. DB
  • Scalability is key many shared data structures
    grow with the number of players, AI, objects,
  • Caution early prototypes may be cheap enough,
    but as game progresses, costs may explode

Why every and all are dangerous
  • Example TSOs Data Service
  • Initial design Transmit entire ReservedLotList
    to all connected clients, every 30 seconds
  • Initial fielding no problem
  • Development testing
  • Complete disaster as clients DB scaled
  • Shipping requirements 100,000 Lots, 4,000
  • LotElementSize LotListSize NumClients
  • 20 Bytes 100,000 4,000
  • 8,000,000,000 Bytes, TWICE per minute!!

Why every and all are dangerous
22,000,000 DS Queries! 7,000 next highest
Costs of QA
  • 6 player game you need at least 6 testers at
    the same time
  • Testers
  • Consoles, TVs and disks
  • Network connections
  • End-user grade network connections (ADSL)
  • More players - more likely to see hard-to-find
    bugs so they are marked as online bugs
  • Automated testing

Debugging Trials Tribulations
  • MP games require MP inputs to test
  • Often, sufficient machines QA testers not
  • Developers significantly handicapped
  • Non-determinism is a serious issue
  • Running the same test twice does NOT necessarily
    give the same result twice
  • Offline code changes frequently breaks online
    code functionality
  • Run-time debugging of networked games often
    becomes post-mortem debugging
  • What helps
  • Automated testing
  • Architectural support for forcing causality (but
  • Strong isolation of online / offline code

Debugging Trials Tribulations
Simple TSO Test One avatar holds the Lot open,
while 3 Avatars jump in out
Debugging Trials Tribulations
Debugging Trials Tribulations
  • TSO EnterLot 30 test runs, 4 behaviours
  • Successful entry
  • Hang or Crash
  • Owner evicted, all possessions stolen
  • Random results observed in all major features
    very difficult to track
  • What worked
  • Massive repetition of tests to establish true
    pass/fail conditions
  • Continual repetition of unit tests (monkeys
  • Initial calibration tests 400x runs per unit
  • 7 to 30 failure of any given unit test

Debugging Trials Tribulations
Monkeys Continual Repetition of Critical Path
Unit Tests
Impact On The Development Team
  • Changes to non-networked gamplay often break
    networked play (shared code implicit
    assumptions timing changes)
  • Changes to online code can inhibit content
    development (server down often equals nobody
  • Non-determinisim produces frustrating defects
  • Mitigating factors
  • Make sure people can work in isolation no
    critical path failures that bring down the team
  • Strong testing in the dev team
  • Online/offline code sharing via modules, not

Stability Analysis What Brings Down The Team?
Test Case Can an Avatar Sit in a Chair?
Failures on the Critical Path block access to
much of the game. Worse, unreliable failures
use_object ()
buy_object ()
enter_house ()
buy_house ()
create_avatar ()
login ()
Impact On Others
Process Shift Comb Filter Testing
Sniff Test, Monkey Tests - Fast to run -
Catch major errors - Keeps coders working
Smoke Test, Server Sniff - Is the game
playable? - Are the servers stable under a
light load? - Do all key features work?
Full Feature Regression, Full Load Test - Do
all test suites pass? - Are the servers stable
under peak load conditions?

New code ready For checkin
Promotable to full testing
Promotable to paying customers
Full system build
  • Cheap tests to catch gross errors early in the
  • More expensive tests only run on known
    functional builds

Lunch Break until 200
  • 200pm 400pm
  • Case Studies

Break until 430
  • 430pm 530pm
  • Post Ship Issues

Cheating The Big Picture
  • A few serious questions every developer should
  • Why do we care if anyone cheats at our game?
  • Is it really a problem?
  • What is our relationship with our customers?
  • What are we willing to do about it?
  • What about our publisher?

Who is cheating?
  • Anyone who can get away with it
  • Profiling is valid here Young males are at
    greatest risk.
  • We have a win at all costs culture that does not
    value honesty
  • The Anonymous nature of the net allows people to
    step outside their normal behavior limits
  • Single player cheats have not helped

Why are they cheating?
  • To Win
  • To cause others not to win
  • Crashing others is considered a win
  • To exercise control
  • Almost pathological need for some
  • To show off (attention)

Matts Rules about Cheating
  • Rule 1 If you build it, they will come -- to
    hack and cheat.
  • Rule 2 hacking attempts increase with the
    success of your game.
  • Rule 3 Cheaters actively try to keep developers
    from learning their cheats.
  • Rule 4 Your game, along with everything on the
    cheater's computer, is not secure. The files are
    not secure. Memory is not secure. Services and
    drivers are not secure.

Rules, cont.
  • Rule 5 Obscurity is not security.
  • Rule 6 Any communication over an open line is
    vulnerable to interception, analysis, and
  • Rule 7 There is no such thing as a harmless
    cheat or exploit. Cheaters are incredibly
    inventive at figuring out how to get the most out
    of any loophole or exploit.

More Rules
  • Rule 8 Trust in the server is everything in a
    client-server game. Actually its true for all
    game types.
  • Rule 9 Honest players would love for a game to
    tip them off to possible cheating. Cheaters want
    the opposite.
  • Rule 10 No one likes to have their ass handed
    to them on a platter every time

Rules, cont.
  • Rule 11 Once a cheat has been perpetrated, a
    patch, sequel, or similar game will suffer
    similar hacking attempts sooner than later.
  • Rule 12 There is more than one way to
    perpetrate a cheat.
  • Rule 13 It only takes one person to break an
    online community.

The Last Rule
  • Rule 99 There is no Silver Bullet to make
    your games vulnerabilities go away. It is an
    ongoing, never ending war. What a developer can
    do is to choose which battles are worth fighting.

Reflecting on the rules
  • What is the purpose of a computer game?
  • To entertain create enjoyable experiences
  • To make the developer money
  • To make users desire your future products
  • Who owns the game experience?
  • What exactly did the player buy?
  • What are the obligations and expectations?

General Categories of Cheats
  • Reflex augmentation
  • Authoritative clients
  • Information exposure
  • Compromised servers
  • Bugs and design loopholes (exploits)
  • Environmental weaknesses
  • Social Cheats

Reflex Augmentation
  • Anything that replaces human skill response
    with automated input.
  • FPS Aiming proxies or bots.
  • Situation analyzers or AI players.
  • Timing assistants.
  • Changes the game from a contest of two humans to
    one of man vs. machine.

Authoritative Clients
  • When a players client is the final arbitrator of
    any aspect of the game
  • Less of an issue for server based games
  • Obviously someone has to make some definitive
    decisions in a peer to peer game.
  • Can all players make the same decision and then
    compare their conclusions?

Information Exposure
  • The revealing of any game information hidden from
    the player
  • More of a problem for synchronous simulations
  • Can be totally passive
  • RTS Map Fog of War hacks
  • Skin model hacks
  • View of other players status

Compromised Servers
  • Related to general PC security issues
  • More likely done by the server operator, but
    could be done by someone else
  • A perceived problem with clans
  • How to detect?
  • Ties into matchmaking issues.

  • Coding bugs and design loopholes
  • Catchall category of problems
  • Often discovered by accident
  • Question of who makes the call

Environmental Weaknesses
  • When the games environment is subverted
  • Corrupted Communication data
  • Hacked Modified Drivers
  • OS bugs
  • Hacked System Clocks
  • Bandwidth and latency calculations

Social Cheats
  • Social Engineering that takes place inside of a
  • Online forms of issues and scams weve had in
    person for thousands of years.
  • Player Collusion and betrayal.
  • Player in the same physical location.
  • Player controls pace of game abusively
  • Player text/voice chats others abusively

Other problems Identity Theft
  • The impersonation or hijacking of a players
    online identity and legacy.
  • More of problem as we push MMO games, persistent
    games and meta-games.
  • Players often have a lot invested in their
    on-line avatars and identities.

Other Reverse Engineering
  • Reverse engineering of Game Rules and Stats
  • This is a design issue
  • Non published Statistics
  • Random Number Generators

Some Grey Areas
  • Engine Game Configurations
  • Separates the hardcore player from the causal
  • Connection Type
  • Not much we can do.
  • Equipment Configuration
  • Ditto for PC

Grey Areas cont.
  • Cheaters vs. Impossibly Good Players
  • What happens if someone cant tell the
  • The skill curve of your game
  • Can be prohibitive to new players

What the developer can do
  • Plan for anti-cheating efforts in the schedule.
  • Audit your (final) communication formats.
  • Audit your data program layout.
  • Attempt to make cheats using your insider
  • Plan for a patch.
  • Provide auditing tools to your players.
  • Create persistent player identities.

What developers can do, cont.
  • Know in advance the public stance you and your
    publisher will take if a cheat is found.
  • Denial can work, but is risky
  • Honesty is appreciated by players
  • But it could be an admission of liability?
  • Communicate with your players
  • Active community management.
  • Cultivate contacts in the user community

How long do you support a game after it ships?
  • Problem no available staff
  • Problem It costs
  • One solution metric based on sales
  • Make a decision, be willing to set an EOL date
    for support?

Dealing with Authoritative Clients
  • Run the full simulation on all machines
  • Communicate only requests, not actions
  • Harder than you think
  • Assume nothing regarding game rules
  • Verify the full state of the simulation
  • Great debugging benefits

Combating Information Exposure
  • Add detection processes to client.
  • Share information on the engines display
  • Good for complex data
  • Simple data can still be passively extracted
  • Audit previous gameplay for information
  • Use Honeypot designs.

Dealing with Compromised Servers
  • Give the users tools to determine the servers
    state configuration.
  • Link Identities to operators.
  • Allow users to filter operators
  • Use third party products like Punkbuster or

Dealing with Exploits
  • Patch, baby, patch.
  • Most effective patch case
  • Have a consistent policy regarding changes
  • And document the changes to your users
  • changes in patches not communicated can be seen
    as new cheats or bugs.

Other things developers can do
  • Develop an internal cheater team from your QA
    team during the development process.
  • Provide them with assistance to crack your own
  • Provide Tools to combat Social Cheating
  • Recorded games and auditing tools

Programming Recommendations
  • Assuming that your game will be hacked for the
    purpose of cheating.
  • Make an effort to determine the likely forms that
    the cheat will take, and put code in your game
    whose sole purpose is to detect evidence that a
    cheat has taken place.
  • For very likely cheats, place detection code at
    multiple levels in the game if possible.
  • Once cheating has been detected, delay responding
    if possible.

More recommendations
  • Single player cheats need to go in their own
    separate game system.
  • Have your game validate the integrity of its
    executables, dynamic link libraries, and
    non-variant data files

Check your output!
  • Dont leave debug symbols or developer
    functionality compiled into the games executable
  • Always scan your games executable files for
    strings and other revealing data before releasing
    the files to anyone outside of the development
  • Make sure that all debugging and developer code
    is actually not compiled in external release
    versions of your game.

Dont forget to check data files too!
  • Make sure that all developer and debugging
    information and functionality is removed from the
    data files that accompany the games executables.
  • If your game uses a scripting language or other
    data system that exposes human readable function
    or command names, implement a tokenizer or
    compiler that will change that data to a
    non-obvious form, and dont release the
    uncompiled data files.

More thoughts about cheating
  • Never underestimate the techniques hackers will
    use to rip apart a game.
  • Once a code-related hack has been created, fixing
    it with a patch can be a no-win situation.
  • Get it right the first time when you patch.
  • Consider performing radical changes to the games
    code to fix a hack.
  • Previous versions of a game provide a source of
    difference data for a hacker.

Patching Multiplayer Games
  • Patching on consoles ranges from impossible, to
    very difficult, to only requiring a
    certification pass.
  • Who pays for the patch to be built?
  • How do you get it to your players?
  • Network budget
  • Impact on the player experience
  • Differential copies game changes to simplify the
  • What conditions require a patch
  • How bad do things have to be to require a bug
  • Or do you patch to deliver new content?
  • Remember that any patch has a customer service
  • Remember that the nature of a multiplayer game
    requires the players to patch if they want to
    play with others.

Long-term customer support
  • Making support calls not happen must be a
    priority. Especially for games without a
    recurring income stream, any CS call may put you
    cash negative on the sale.
  • Poor customer support can drive customers away
  • However, many customers may have unreasonable
    expectations and they will never be satisified,
    regardless of the CS quality.
  • Official forums tend to be extremely negative.
    Happy players usually dont post, and unhappy
    players want everyone to know they are unhappy,
    and will post in the only forum they know the
    developer/publisher reads.

Cross platform support
  • Risks All platforms that are interoperable must
    be able to be patched roughly simultaneously
  • Risks Small differences in floating point
    processing can give an unintentional advantage to
    some platforms
  • Risks Depending on the differences between
    platforms, input devices may give a significant

New Content
  • MMP New content must be added, constantly
  • Strong impact on old content
  • Gameplay balances, long term additive properties
    (N2ed? Worse?)
  • Code rot, tools, and long term maintainability
  • Free vs user created vs commercial expansions
  • Legal issues
  • Distribution installation with N prior
  • Gameplay imbalances
  • Providing hooks for player generated content is
    not cheap!

Open QA
  • 530pm - 600pm

  • References available as part of the final slide
  • Final Slides available online _at_