Wake up and smell the metrics Metrics Driven Development in online games

1
Wake up and smell the metrics!Metrics Driven
Development in online games

• Larry Mellon
• Darius Kazemi

AGC 08
2
Data mining is pure gold!What stops people from
reaching it?
3
Cultural collision is our Smaug!
Boxed goods mentality
Online service reality
Wrong assumptions lead to painful decisions!
4
Larry Mellon(Consultant)
Research era

• Alberta Research Council Jade Simulations
• Distributed computing, 1982
• Optimistic computing, 1000 CPU virtual world
• Fault-tolerant cluster computing
• Synthetic Theatre of War virtual worlds for
  training
• DARPA 50,000 entities in real-time virtual
  worlds
• ADS, ASTT, HLA RTI 2.0, interest management

Wife era

• EA (Maxis) The Sims Online, The Sims 2.0
• Scalable server architecture
• Automated testing to accelerate production and QA
• Player, pipeline performance metrics
• Emergent Game Technologies (CTO)
• Architect for scalable, flexible MMO platform

5
Darius Kazemi Orbus Gameworks darius_at_orbusgamework
s.com
6
Boxed goods industry is evolving online and
needs to adapt to service industry requirements!
7
Why is it hard to get funding for something as
clearly useful as metrics?
Nobody wants to pay for it, because no one has
traditionally paid for it!
?


Features
QA
Metrics, CS,
8
Diasporas kill off the home-rolled systems
9
Business Intelligence has driven the success of
many other industries for years!
Las Vegas Strip
10
Metrics help manage complexity
When you can measure what you are speaking about
and can express it in numbers, you know something
about it. But when you cannot measure it, when
you cannot express it in numbers, your knowledge
is of a meager and unsatisfactory kind." - Lord
Kelvin Institution of Civil Engineers, 1883
The general who wins the battle makes many
calculations in his temple before the battle is
fought. The general who loses makes but few
calculations beforehand. -- Sun Tzu
11
The people who led the scientific revolution
depended on metrics!
Pasteur A science is as mature as its
measurement tools. Maxwell To measure is to
know.
12
The three largest factors that will influence
gaming will be and metrics (measuring what
players do and responding to that) -- Will
Wright The Secret of The Sims", PC Magazine,
2002. http//www.pcmag.com/article2/0,1759,482309
,00.asp
13
GIGO Avoid false causality by correlating
data!
14
Multiple views of data provides a deeper
understanding and fewer analysis errors
Time
Minute two
Minute one
15
3 Final road block Metrics are role-specific
and hard to move across boundaries
16
The 3P's model of game metrics

• Player
• Performance
• Process

17

• Next Gen Games
• Increased Complexity
• Increased Complexity of Analysis

Art from Fun Meters for Games, Nicole Lazzaro
and Larry Mellon
18
(No Transcript)
19
Next Gen Games
20
Tuning imbalances or exploits can throw your
entire economy out of kilter, but remember to
triangulate!
21
Tuning imbalances or exploits can throw your
entire economy out of kilter, but remember to
triangulate!
22
Know Your Players
23
Comparing groups is another no-brainer
24
Process metrics

• Find the leaks that are slowing you down or
  costing you money!
• Another cultural problem
• Process evil
• Tools ! game feature
• Not fun to build
• No status
• Thus, worst possible programmers are assigned to
  most critical systems

25
Fixing development leaks is like adding free
staff!
26
Process Measurable targets projected trends
give you actionable progress metrics, early
enough to react
Target
Risk!
Any metric (e.g. clients)
Time
Any Time (e.g. Alpha)
27
Real data gives you faster focus, less debate!

• Question for the audience
• If you have 5 engineers in the room, how many
  opinions do you have on what the team needs to do
  next?
• Numbers cut the debate from what is next? to
  how do we fix the thing we found?
• You QUICKLY go from finding the key data and
  fixing the problem, not searching the entire data
  space, just to get started!

28
Performance engine metrics(Dev Ops)

• Debugging optimizing large-scale, complex
  system
• With hackers, non-determinism constant change
• Tick-tock!! Users (and VPs) are waiting
• Most engineering metrics tools are for one
  person, one machine, one debugger
• Need system-level view to deal with MMOs
• Studio head metric only team-wide, red-ink email
• GET THIS METRICS SYSTEM IN THE LIVE SERVERS, BY
  TOMORROW (tired of black box debugging across
  20 servers and 200 clients)

29
Lower dev ops costs FREE MONEY!
Profit
New Content
Regression
Customer Support
Operations
30
Profit
Lower New Content Cost
Regression
Customer Support
Operations
31
Profit
Lower New Content Cost
Lower Testing Cost
Customer Support
Operations
32
Profit
Lower New Content Cost
Lower Testing Cost
Happy Customers Dont Call
Operations
33
Process (Dev Ops) Lower recurring costs that
are worth the cost to lowermakes you more
money with the same number of players!
Profit
Lower New Content Cost
Lower Testing Cost
Happy Customers Dont Call
Operations
Lower bandwidth CPU
34
Process Rapid Iteration is how you find fun!
Project Start
Launch
35
Implementation options(both are hard)

• Grassroots movement scale as required
• DEFINE metrics as a fully funded feature needed
  for success in an online service!

36
1.a How to start your own Mini metrics project
Image by Flickr user RaeA Creative Commons
(BY-NC-SA)
37
1.b How to scale your own Mini metrics project
Image by Flickr user Dredrk Creative Commons
(BY-NDW)
38
Culture Shift optionTreat metrics as a critical
feature from day one!
Fund everything that helps both team and
customers, not just game play!

!!!

Features
QA
Metrics
39
Metrics are your first-pass tool they quickly
find you a starting point that would take
hours/days for you to find by yourself
40
Find oddities automatically for you, then you
decide what action to take
41
Scaling the metrics system as data scales
Tip make your system self-maintaining, and
critical to programmers and/or management (theyll
fix it the moment it breaks ? )
42
Train bots to look for patterns of data when the
datasets get even larger
43
Takeaways

• Grassroots or formal feature either way, get
  started with metrics across your entire game
  service
• Find something that will help an obviously
  important question to be answered with facts
  rather than guesses
• To get executive and funding support, translate
  your day-to-day need for metrics into the metrics
  the executives understand to justify the funding
• You will either make more cash with this change!
• Or youll save more cash with this change!
• Presentation and aggregation are key use the
  computer to do the hard parts
• People should see high-level data, with easy
  search and correlate functionality, and access to
  the raw data

44
The migration online is a Darwinian moment for
our industry

• Boxed goods mentalities must shift to online
  service realities, or die
• Complexity people need metrics to be able to do
  their jobs
• Player Retention Its not just about game
  features anymore!

45
QuestionHow would you rather live your life?
Measure
Change
Measure
OR
Guess
Change
Hope
Slides are online at http//www.maggotranch.com/bi
blio.html www.orbusgameworks.com/blog