Displaced SoundScapes

1
Displaced SoundScapes

• Álvaro Barbosa abarbosa_at_iua.upf.es
  porto.ucp.pt
• Music Technology Group, Audiovisual Institute
• Pompeu Fabra University Barcelona, Spain
• Department of Sound and Image
• School of Arts Portuguese Catholic university

2
MOTIVATION

• So far most systems explore communication via
  computer networks, for Music and Sonic Arts
  applications, approaching the possibilities of
• Creating networks of users in which digital audio
  or symbolic documents can be exchanged amongst
  contributors
• Providing a channel for Tele-presence or remote
  collaboration between performative spaces

3
MOTIVATION

• The shared nature of the Internet suggests a
  different approach
• To create community oriented Shared Virtual
  Environments (SVEs) in the context of Music and
  Sonic Arts.
• Like in similar paradigms, oriented towards
  visual or textual communication, this concept
  tends to lead towards new mechanisms of
  interaction not usually found in a musical
  context. (Curtis, P. 1992)
• It is an open area of research with many
  questions to be addressed.

4
MOTIVATION

• What is the role of these systems creators?
• Is community music an aesthetical meaningful
  Sonic Arts form?
• How do the sonic results of these systems fit
  into time and space as we know it in the musical
  context?
• Is the regular internet user ready to go beyond
  the role of a spectator and become a creator,
  composing, performing or improvising in a music
  piece?
• What kind of constraints can, or should, be
  considered in the user interaction layer?
• Is Multimodal interface design a possible
  approach for community music?
• Should one aim for a visual environment
  interaction models that are driven by the
  Soundscape? Or the other way around?
• What should be handled by the user and what
  should be handled by the system?
• Could there be general-purpose models, at the
  architectural and acoustic communication level,
  that address the specificities of this paradigm?

5
RESEARCH WORK

• Preliminary research on this topic was conducted
  in the context of the authors Doctorate studies
  at the UPF/MTG and was approached in four parts
• A systematic study of Group Communication in
  Computer Networks and general systems for
  Computer-Supported Cooperative Work
• Overview of the state of the art in collaborative
  systems for Music and Sonic Arts applications
• Analysis and discussion of the experimental
  project Public Sound Objects
• Proposals for general Architectures and Models

6
1. GROUP COMMUNICATION IN COMPUTER NETWORKS
7
1. GROUP COMMUNICATION IN COMPUTER NETWORKS

• General study and discussion of
• Common Network Protocols
• (TCP, UDP)
• Reliability and Quality of Service
• (latency, jitter, packet loss and de-sequencing)
• Network Communication Models
• (centralized/distributed network model and the
  group communication paradigm)
• Decentralized Communication Environments
• (peer-2-peer and mixed topologies)

8
1. SYSTEMATIC STUDY OF CSCW SYSTEMS

• Computer-Supported Cooperative Work
• CSCW term was introduced in 1984 by Irene Greif
  from MIT and Paul Cashman from Digital to
  describe the object of interest for a small
  workshop about computer systems which would
  support people in their work activities (Bannon,
  J. 1992).
• Today CSCW is recognized as a pertinent field in
  Computer Science and represent a major focus area
  of the Special Interest Group on Groupware
  (SIGGROUP), from the ACM Organization.
• It is multidisciplinary field which brings
  together people from different backgrounds like
  computer science, psychology, sociology,
  organisational theory, and anthropology, just to
  mention a few.

9
1. SYSTEMATIC STUDY OF CSCW SYSTEMS

• Computer-Supported Cooperative Work
• 1992 Roddens Survey on CSCW systems introduces
  different operation modes
• Message systems (ex. Email based systems)
• Computer Conferencing (ex. Video/Audio Real Time
  conference)
• Coordination Systems (ex. White boards)
• Co-Authoring Systems (ex. Software development
  systems with updated version tracking)
• And different environmental Facets
• Synchronous / Asynchronous
• Co-Located / Remote

10
1. SYSTEMATIC STUDY OF CSCW SYSTEMS

• Shared Virtual Environments
• Going beyond CSCW by creating a shared space in
  a computer network, inside which users can
  achieve a certain degree of immersion and
  flexibility in their behaviour.
• MUD Early 80s by Richard Bartle and Roy Trubshaw
  at the University of Essex's in England
• Moo (MUD Object Oriented) by Pavel Curtis 1996
  Xerox Park
• Habitat game developed by Lucas Film in 1985
  (introduces the concept of Avatar)
• Active worlds and DIVE use Sophisticated 3D, and
  the Habbo Hotel uses simple 2D graphics and its
  quite successful since January 2001, and many
  others.

11
2. COLLABORATIVE SYSTEMS FOR MUSIC AND SONIC ARTS
12
2. COLLABORATIVE SYSTEMS FOR MUSIC AND SONIC ARTS

• The idea of musical computer networks is not new
• Local Interconnected Musical Networks (INMs)
• Early implementations (although on a local area
  scale) date back to the late 1970s with
  performances by the League of Automatic Music
  Composers (Bischoff, Gold and Horton, 1978).

• First experimental systems go back to the early
  1990s with the NetJam System developed at
  Berkley University. This system allowed a
  community of users to collaborate producing music
  in an asynchronous way by automatically
  exchanging MIDI files trough internet e-mail
  (Craig R. Lattas, 1991).

13
2. COLLABORATIVE SYSTEMS FOR MUSIC AND SONIC ARTS

• Typical Examples of musical collaboration over
  the Internet
• Project oriented collaboration paradigms
• 1994, The ResRocket Surfer - Rocket Network M.
  Moller, W. Henshall, T. Bran and C. Becker.
  (Video - rocketnetwork.mov)
• 1999, Faust Music On Line (FMOL) S. Jordà.
• 2001 Tonos - Online Musician's Network TC8 -
  Music Collaboration Tools (8 track recording
  Software, which allows users to use a centralized
  account with 40Mbytes of Hard-Disk Space).

FMOL Client Interface
FMOL On-Line Data Base
Tonos-TC8 Client Interface
14
2. COLLABORATIVE SYSTEMS FOR MUSIC AND SONIC ARTS

• Typical Examples of musical collaboration over
  the Internet
• Tele-Presence between performative Spaces
• 1996, The Cassandra Project D. Ghezzo, J.
  Gilbert, A. Smith and S. Jacobson (NYU - Simon
  Fraser University Vancouver).
• 1999, Piano Master Classes via de Internet J.
  Young and I. Fujinaga.
• 2000, Real-Time Streaming of Multi-channel Audio
  Data over Internet. Demonstration concert at the
  108th Convention of the Audio Engineering Society
  Robert Rowe, Jeremy Cooperstock.
• 2000 Montreal-Ottawa Teaching Trial (190 Km, 40ms latency) using Internet-2 over 1Gbps
  connection (SDI Video on a 50 plasma Screen and
  Multi-Channel 96KHz, 24bits audio), Pinchas
  Zukerman and Veronique. (Video - Zukerman1.mov,
  Zukerman2.mov)
• 2000 New York-Ottawa Remote Master Class (540 Km,
  60ms latency) with mpeg2 compression at 10Mbps,
  Pinchas Zukerman and Wu Ji. (Video 3)
• Milestones in Real Time Networked Media at Jeremy
  Cooperstocks Website http//www.cim.mcgill.ca/j
  er/research/rtnm/history.html

15
2. COLLABORATIVE SYSTEMS FOR MUSIC AND SONIC ARTS

• Collective Creation or Remote Collaboration
• Public on-line community involved in major events
• 1996/2000 Brain Opera Brain Opera session in
  Viennas Haus der Musik Tod Machovers (on-line
  users intervention at the events by actuating on
  a web client).
• 1999 Faust Music On Line (FMOL) Faust Opera by
  La Fura Dels Baus in Barcelonas Teatro del
  Liceu S. Jordà.

16
2. COLLABORATIVE SYSTEMS FOR MUSIC AND SONIC ARTS

• Collective Creation or Remote Collaboration
• Peer to Peer synchronous collaboration
• 1997 Transmidi MIDI based synchronous
  interaction network system, where users can play
  together in sessions performing on their MIDI
  controllers/instruments D. Gang.
• 2000 TransJam MIDI and Digital Audio (low
  quality) synchronous network system for jamming
  over the internet Phil Burk.
• 2003 PeerSynth Real time P2P network
  communication through TCP/IP. High resolution
  musical parameters (not audio data) are sent and
  received in real time by the users software
  instances Jörg Stelkens.

TransMidi System and Possible Topologies

PeerSynth Interface
17
2. COLLABORATIVE SYSTEMS FOR MUSIC AND SONIC ARTS

• Net Music what to do with network latency?
• Internets latency can be seen as cyberspaces
  acoustic (Atau Tanaka)
• Physical Model Synthesis with Application to
  Internet Acoustics (Chris Chafe)
• Network latency is a technological condition that
  tends to be reduced to the point of not being an
  impediment for real-time music communication at
  medium distances (Broad Band connectivity, data
  compression, content based transmission)
• However at a global level it will always exist.
• For communication between two opposite points of
  the planet considering a best case scenario
  (light speed connection over the earth surface),
  the round-Trip Latency is about 42 ms.

18
2. COLLABORATIVE SYSTEMS FOR MUSIC AND SONIC ARTS

• Latency Tolerance in Jazz Music
• An experiment was conducted by the author during
  the months of June and July of 2004.
• 4 musicians (Bass, Guitar, Piano and Percussion)
  performing Jazz Standard Tunes in duets
• Studio simulated Network delay conditions on the
  acoustic interaction.
• From this experiment resulted over 100 hours of
  recordings whish will be used in a user study to
  evaluate the effects of latency in jazz duets.

19
2. COLLABORATIVE SYSTEMS FOR MUSIC AND SONIC ARTS
Latency Tolerance in Jazz Music Several
Feed-Back setups, delay and tempo conditions
where tested including a Self-Delay Tolerance
test.
Preliminary results show that the latency
tolerance decreases with latency. The Higher the
Delay between musicians the slower must be the
tempo of the song to maintain synchronization.
20
2. COLLABORATIVE SYSTEMS FOR MUSIC AND SONIC ARTS

• Shared Sonic Spaces on the Internet
• Community oriented Shared Virtual Environments
  (SVEs) in the context of Sonic Arts.
• An aspect of sonic collaboration over the
  internet, that goes beyond the idea of enhancing
  the established acoustic communication paradigms.
• Issues can be raise regarding the time scale of
  shared events on the internet (permanent
  events???)
• Examples of existing systems The WebDrum (Burk,
  P. 2000), MP3Q (Tanaka, A. 2000).

21
2. COLLABORATIVE SYSTEMS FOR MUSIC AND SONIC ARTS
Classification Space for network music
systems Based on Roddens environmental Facets
(Rodden, T. 1992)
22
3. THE PUBLIC SOUND OBJECTS PROJECT
23
3. THE PUBLIC SOUND OBJECTS PROJECT

• The Public Sound Objects project and its context
• Focus on Internet Collaborative Virtual
  Environments
• for music and sonic arts.
• A Web based Shared Sonic Environment and
  simultaneously an on site sound installation.
• Connected users can control a server side
  synthesis engine through a web-based interface.
• Each users input has a direct and immediate
  influence on the resulting soundscape, by the
  means of a Sound Object, as defined by Pierre
  Schaeffer in the 60s Any sound Phenomenon
  perceived as a coherent whole () regardless of
  its source or meaning .
• The resulting sound is streamed back to each user
  and is also played back at the installation site.

24
3. THE PUBLIC SOUND OBJECTS PROJECT

• An ongoing Public Event in the internet
• Most public artistic proposals designed for
  community performance have been developed towards
  an event that takes place at a specific date
  during a certain period of time, in the presence
  of a physical and/or virtual audience in a
  theatre-like experience.
• Internet provides permanent connectivity, thus a
  public Internet event can go on permanently.
• In this system the audience and performers will
  be able to dynamically join and leave the
  performance at any time they want.
• (Video - PSO-System.mov)

25
3. THE PUBLIC SOUND OBJECTS PROJECT

• System Architecture
• In the Preliminary Prototype the interaction
  feedback will be auditory only and it will convey
  the performance of all the connected users at a
  given moment with the respective latency due to
  network speed limitations.
• In the Public installation Site the soundscape
  will have added spacialization, and visual
  representation of the users performance.

26
3. THE PUBLIC SOUND OBJECTS PROJECT

• The Synthesis Engine
• The Synthesis Engine for the prototype is
  developed in PD (Pure Data).
• The synthesizer provides a sound generator which
  ranges from basic oscillators up to the use of
  digital sound samples.
• This generator's waveform is then altered by a
  chain of modifiers, providing the transformed
  sound object to be included in the resulting
  Soundscape.

27
3. THE PUBLIC SOUND OBJECTS PROJECT

• The Interface Layer
• The user to contributes to the musical
  performance by transforming the characteristics
  of a visual Sound Object representation, sending
  normalized parameters over the network to the
  synthesis engines transformers.
• The GUI should be a behaviour driven metaphorical
  interface, avoiding a flat mapping of parameters
  (inspired by Kiyoshi Furukawas Small Fish).
• The Sound Object visual representation has an
  automatic periodical behaviour that is conducted
  by the user.

The ball is moving continuously and the user can
manipulate its size (1), its speed and direction
(2), each walls acoustic texture - Pitch (3),
Ball Tail -Delay (4) and Ball Speed Reduction
According to Network latency towards the server
(5).
28
3. THE PUBLIC SOUND OBJECTS PROJECT
The Public Installation Site
29
3. THE PUBLIC SOUND OBJECTS PROJECT
The Public Objects Trial Concert and
Installation From the 7th to the 13th of
October the system was fully installed with
on-line and on-site setups. During this trial
period it was tested by over 200 users and a
concert, featuring 6 simultaneous users, was
performed. (Video - PSO-Concert.mov)
30
3. THE PUBLIC SOUND OBJECTS PROJECT
The Public Objects Trial Concert and
Installation For this trial experiment the
interface was adapted for different platforms
(regular computer with mouse and keyboard
interface, Touch-Screen and PDA) in an
installation with wireless network access
(sponsored by KPN-Quest).
31
3. THE PUBLIC SOUND OBJECTS PROJECT

• The design principles and concept for this
  collective composition environment are now quite
  clear.
• The first system prototype is now working for
  nine simultaneous Sound Objects, and it has a
  good feel of playability even at highly remote
  locations, provided by the network adaptive
  tempo(speed).
• http//www.iua.upf.es/abarbosa/ and
  http//soundserver.porto.ucp.pt/pso
• From the trial installation in October 2004, over
  a 100 inquiries where obtained from system users,
  however the analysis and evaluation as not yet
  been done.
• Future versions will include the possibility to
  upload user files and an optional client
  synthesis engine to provide local individual
  feed-back and individual sound level control.

32
4. PROPOSALS FOR GENERAL ARCHITECURES AND MODELS
33
4. PROPOSALS FOR GENERAL ARCHITECTURES AND MODELS

• Interaction Model for an individual User
• Virtual Music Instrument VS Decoupled VMIs

34
4. PROPOSALS FOR GENERAL ARCHITECTURES AND MODELS
General Purpose Centralized Architecture
35
4. PROPOSALS FOR GENERAL ARCHITECTURES AND MODELS
General Purpose Mixed Architecture (P2P
Centralized DB)
36
CONCLUSIONS AND FUTURE WORK

• It is clear Shared Sonic Environments are quite
  an extremely promising area for research, not
  only by the fact that it allows the enhancement
  of known paradigms to make music, but also
  because it provides a context for stylistic
  novelty.
• This Research work conveys relevant references
  and concepts for future work in this field.
• Future work plans include further developments in
  the PSOs Project and in the ReactTable Project,
  both in the framework of the Interactive Systems
  Group at the MTG.

37
PUBLISHED ARTICLES

• Barbosa, A. 2004 "Displaced Sondscapes over W3"
  W - Art Nº 003, Contemporary Art Magazine 2004,
  Nemesis Porto, Portugal.
• Barbosa, A. 2003 Displaced Soundscapes A Survey
  of Network Systems for Music and Sonic Art
  Creation Leonardo Music Journal 13 - MIT
  Press, Cambridge MA.
• Barbosa, A. Kaltenbrunner, M. 2002 Public Sound
  Objects A shared musical space on the web.
  Proceedings of International Conference on Web
  Delivering of Music 2002 Darmstadt, Germany,
  published by the IEEE Computer Society Press.
• Jordà, S. Barbosa, A. 2001 Computer Supported
  Cooperative Music Overview of research work and
  projects at the Audiovisual Institute UPF.
  Proceedings of MOSART Workshop on Current
  Research Directions in Computer Music 2001
  Barcelona, Spain.

38
Acknowledgements

• Álvaro Barbosas Thesis Advisor Xavier Serra
  (MTG Barcelona)
• PSO Developers
• Jorge Cardoso, João Seabra UCP Porto
• Gunter Geiger, Martin Kaltenbrunner MTG
  Barcelona
• Performers and staff at the PSO Concert
  (UCP-Porto)
• Álvaro Barbosa, Carlos Caires, João Seabra,
  Jorge Cardoso, Marta Reis, Rodrigo Areias,
  Mariana Madail, Mafalda Barbosa.
• Latency Tolerance experiment (UCP-Porto)
• Alexander Carôt, João Seabra, Nuno Archer, Paulo
  Gonçalves
• Thank You!!!!