Positional Audio and Window Highlighting for the Access Grid

1
Positional Audio and Window Highlighting for
the Access Grid

• Robert Putnam
• (putnam_at_bu.edu)
• Boston University
• Scientific Computing and Visualization

2
Agenda

• The problem
• Prototype positional audio
• Prototype window highlighting
• Distribution/installation
• Discussion

3
Who's talking now?

• In some AG contexts, it is not always clear which
  video window contains the current speaker. The
  problem is exacerbated by
• Multiple, concurrent speakers
• Users unfamiliar with the AG
• Users unfamiliar with each other
• Remedy look for familiar face or moving lips.

4
Whats that noise?

• In some AG contexts, it is not always clear who
  is transmitting echo or noise.
• Remedies
• Broadcast appeal
• Operator mutes RAT streams one at a time until
  offender is found.

5
Solution 1 Positional audio

• Use location of site's video window(s) to
  determine its audio position in stereo field.

6
Solution 1 Positional audio

• Preliminary system design
• Components
• (Minimally modified) vic and RAT
• Agent which communicates with vic and RAT
• Sequence of events
• Operator moves window
• Agent gets report of window movement from vic
• Agent sends audio panning command to RAT
• Need communications audio localization layers.

agent
RAT ?
vic
?
?
7
Positional audio

• RAT has built-in support for 3D audio
• RATs localization methods
• Interaural Intensity Difference (IID)
• Difference in sound intensity between near and
  far ear.
• Interaural Time Difference (ITD)
• Difference in time of arrival between near and
  far ear
• Head Related Transfer Function (HRTF)
• 3D-location-specific filtering effect of ears,
  head, and torso.

Actually 1D, since theres only a single
degree of freedom.
8
Positional audio

• Revised system design
• What about communications? Next slide

agent
RAT 3D audio
vic
?
?
9
Message Bus

• RATs control, user interface and engine already
  communicate via the Message Bus (mbus)
• So

rat-ui
rat-engine
mbus
rat-control
see www.mbus.org
10
Positional audio

• Revised system design
• Next what mbus message is used to enable 3D
  audio?

agent
RAT 3D audio
vic
mbus
mbus
11
3D audio mbus message

• 3D audio mode is enabled for a given RAT source
  via an audio.3d.user.settings message

audio.3d.user.settings
rat-ui
rat-engine
mbus
tcl
see www.mbus.org
12
Talking to RAT

• So, agent sends audio.3d.user.settings message to
  rat-engine
• ssrc Synchronization source RTCP stream id
• azimuth horizontal angular distance of listener
  from sound source. (90 panned left -90
  panned right 0 panned center)

audio.3d.user.settings ssrc azimuth
agent
rat-engine
vic
?
mbus
mbus
13
Listening to vic

• Next, create new mbus message
• window.xy
• Normalized xpos (0 1.0)
• Normalized ypos (0 1.0)
• Name of video window (e.g., LANL(Video))
• Whenever video window is moved, resized, or
  renamed, send window.xy message to agent.

agent
rat-engine
audio.3d.user.settings
vic
window.xy
mbus
mbus
14
Associating vic and RAT streams

• Agent responsible for determining which video
  windows correspond to which RAT streams (ssrcs)
• As of AGtk 2.4, it is assumed that vic and RAT
  streams have the same name
• I.e., RAT LANL corresponds to vic LANL(Video)

agent
rat-engine
rtp.source.name
vic
window.xy
LANL(Video)
04b6910a LANL
15
Solution 2 Window highlighting

• System shown at May 2003 AG retreat
• Feature request visual indication of current
  speaker
• But, danger of distraction/fatigue (i.e. too much
  visual clutter or flashing).

16
Window highlighting

• Change border color of sites video windows when
  its audio rises above a threshold.
• Two new mbus messages
• RAT rat.source.amplitude(source_id amplitude)
• Report each streams peak amplitude 2-3X / second
• vic highlight.window(window_id on_off)

audio.3d.user.settings
window.xy
agent
rat-engine
vic
rat.source.amplitude
highlight.window
17
Window highlighting
18
Window highlighting

• Window stays highlighted during brief pauses in
  audio unless another site begins transmitting.
• Each sites local video windows also highlight
  when RAT transmitting (On Air alert).
• Window border highlight color and width is
  configurable in AG service.

19
Setting up the system

• Install stereo speakers/amplifiers
• Install stereo wiring to and from echo canceller
  (aka the Gentner)

RAT pc
Gentner
amplifier
line out l r
l r
l r
l r
20
Setting up the system

• Create leftright echo cancellation reference

21
Setting up the system

• Remove or disable original video consumer and
  audio services and add two new ones
• HighlightVideoConsumerService.zip
• PositionalAudioService.zip

22
Setting up the system

• Configure HighlightVideoConsumerService

Venue Client
Windows Desktop
23
What next?

• Ready for beta testers! Get it here
• http//scv.bu.edu/putnam/ag-audio/services-2.4.zi
  p
• Zip file contains services for Windows / Linux,
  PIG / multi-computer configurations.
• Request features / report bugs here
• putnam_at_bu.edu