Recreating Reality

1
Recreating Reality Kenneth Young, SCEE
2
How does sound in a virtual world differ from
sound in the real world?
3
The future of gaming is not Donky Kong, it is
these big expansive worlds Mark Rein, VP, Epic
Games, 12/07/06
Technologies like real-time physics, advanced
surround audio systems, and online multiplayer
platforms are also being used to varying levels,
striving to keep a player immersed in a virtual
real world that the game developer has created
for them Michel Kripalani, Games Manager,
Autodesk, Develop Magazine, July 2006
4
This presentation is really about
5

• The brain is amazing

• If youre not musical youre brain dead!

• The brain allows us to create and appreciate art

• The technology which allows us to make games is
  astounding, but the reason we bother at all is
  that, somehow, we can ignore that they are
  totally and utterly ridiculous. Think about it

6

• You have to be a willing victim to become
  immersed in any virtual world

• Game/Player contract ensures that no matter what
  we throw at the player, as long as it is
  explained competently and entertaining, they will
  accept it

• Chions audio/visual contract emphasises that
  sound and image are separate entities which are
  only fused together in the minds of the audience

7

• Chion calls this synchresis (synchronism
  synthesis)

• Synchresis and the audio/visual contract are
  fundamental to the virtual worlds existence

• They permit the use of acousmatic sound - sound
  which is disassociated from its source

8
(No Transcript)
9

• Before we can use sounds in our virtual worlds
  we need to capture them

• Capturing anything involves a transformation

• Microphone choice and location manipulate sound
  before it is even recorded

• Our brains also manipulate sound

• Microphones dont record what our brains hear

10
Why?

• The brain is capable of some impressive
  filtering

• Microphones are not

• Microphones record everything that hits them
  whereas our brains can be quite selective

• These filtering skills are severely diminished
  when listening to a sound recording

11

• e.g. a recording of this presentation would
  include all background sounds, people noise,
  flattened acoustics etc.

• In our virtual worlds we need to perform this
  filtering manually on behalf of our brains -
  mixing

• Real-time dynamic mixing is a relatively new
  frontier and is a high priority in our next gen
  games

12

• Our perception of the real world is so much more
  complex than that offered by even the best
  binaural recording

• Sound and music can be used to represent some of
  this information which is otherwise absent from
  the virtual world

• Music, in particular, is like monosodium
  glutamate for the ears

13
(No Transcript)
14

• The use of music in virtual worlds is
  particularly interesting because it is so
  incredibly abstract

• Our everyday lives are not scored by music

• In a virtual world, music can be used to
  intensify an experience and act as an emotional
  signifier without revealing itself as a
  manipulating device

• This is a very powerful tool but it is in no way
  realistic

15

• Everything Ive covered so far has been
  applicable to the virtual worlds of both film and
  games

• Surround sound is an area where the two are not
  so closely aligned

• In film, surround sound is largely concerned
  with diffuse sound (hence large number of
  speakers)

• Film considers directional surround sound to be
  distracting to an audience

16

• Game cutscenes tend to stick to the conventions
  of films surround sound whereas in-game sound
  uses the surrounds for directional sound

• This diversion is as a result of the voyeuristic
  nature of film versus the participatory nature of
  games

• Whilst some directional cues are of potential
  benefit to the player it is folly to think that
  surround speakers offer any accuracy

17

• Despite this failure to represent the game
  world accurately, the audio/visual contract
  ensures that ao long as this doesnt infringe
  upon the gaming experience it will be accepted by
  the player

• I think our use of surround sound will become
  more sophisticated than just a panning routine

• One area that is evolving is the mixing of
  surround music

18

• If there is nothing realistic about surround
  sound or music in our virtual worlds, why is
  there a debate about whether surround music
  should make overt use of the surround channels?

• Arguments against include distraction and
  precedence

• Anyone who dismisses surround music hasnt
  heard it done well, if at all Rik Ede, Surround
  Evangelist

19

• Dialogue in our virtual worlds is significantly
  different from genuine conversation

• Dialogue is formed of the written language
  spoken aloud. It tends to

• Consist of complete sentences

• Pause at the end of each sentence

• Be well thought through and articulate

• Stay clear of naturalism unless reflecting the
  emotional state of a character

• Be packed full of information

20

• The only thing dialogue shares in common with
  everyday spontaneous speech is that it is spoken
  aloud

• Its quite hard to demonstrate this because we
  are so used to processing dialogue!

21
(No Transcript)
22

• In games, we tend to give the dialogue too much
  to do it carries all the weight of the narrative
  and exposition

• The spoken word is but one method of
  communicating information

• Here, we should try and emulate the real world
  more by taking some of the weight off of dialogue
  and letting other aspects of the virtual world
  tell the story

23
(No Transcript)
24

• In that scene from Never Cry Wolf information is
  also being communicated via

• Exaggerated foley to highlight the Eskimos new
  clothes

• Packing the bullets away and holding the gun

• The howling wind representing the scientists
  anger and confusion

• The Eskimos brand new teeth

25
Summary Conclusion

• The Audio/Visual contract and synchresis allow
  us to create sound and images as separate
  entities then unite them in the mind of the player

• We have to emulate the brains filtering
  capabilities manually as this ability is
  diminished when listening to recorded sound
  emitting from a limited number of speakers

26

• Sound design and music can be used to convey
  information to the player that is felt in the
  real world but not heard

• Surround sound directionality is not as accurate
  as directionality in the real world

• Dont be afraid of surround music, take
  advantage of it!

27

• Dialogue in our virtual worlds is not, and
  should not, be the same as real spoken language

• Our virtual worlds rely too heavily on dialogue
  to tell their stories and convey information to
  the player this needs to change

28

• Sound in our virtual worlds is surprisingly
  different to sound in the real world. But it is
  only through these differences that we can allude
  to reality and therefore create a plausible,
  entertaining gaming experience

• Lets fulfil our half of the audio/visual contract

29
FIN