A Wearable Wireless Sensor Platform for Interactive Dance Performances

1
A Wearable Wireless Sensor Platform for
Interactive Dance Performances

• Chulsung Park and Pai H. Chou
• Center for Embedded Computer Systems

Yicun Sun Department of Arts-Dance
University of California, Irvine PerCom
2006 Presented by Jeffrey
2
Outline

• Abstract
• Introduction
• Related Work
• Platform Design
• Application Example
• Conclusions and Future Work

3
Outline

• Abstract
• Introduction
• Related Work
• Platform Design
• Application Example
• Conclusions and Future Work

4
Abstract

• Reports on recent development of a wearable
  wireless sensor platform
• for interactive dance performances
• At a fraction of a cubic-centimeter in volume
• This platform is truly wearable and scalable in
  forming wireless networks
• Integrated with a wide variety of different
  sensing devices
• It is a real-time monitoring system for
  activities and physical conditions of the human
  body
• Effectiveness of this platform is demonstrated
  with an interactive dance performance

5
Outline

• Abstract
• Introduction
• Related Work
• Platform Design
• Application Example
• Conclusions and Future Work

6
Introduction

• Interactive dance environment
• Live dancers movements are tracked and used to
  steer the synthesis of musical, graphical, and
  other various special effects in real-time
• Available platforms today are not truly wearable,
  scalable, or able to support high-level
  interactivity

7
Major Contributions (1/4)

• Truly Wearable sensor platform
• Eco
• Ultra-compact and low power wireless sensor node
• 648 mm3 without a battery
• 720 mm3 with a battery
• Smallest wireless sensor node in operation
• Other wearable sensor platforms are at least 3 to
  4 times larger

8
Major Contributions (2/4)

• Make Eco nodes form a scalable wireless network
• Adopt ideas proposed for heterogeneous network
  architecture
• Each dancer and sensor device can be uniquely
  identified
• While multiple dancers wearing multiple sensing
  devices perform together

9
Major Contributions (3/4)

• Novelty in multi-modal sensing
• Can collect data from multiple different types of
  sensing devices simultaneously
• Motion tracking on dancers
• Reading their physiological signs such as
  heartbeat
• Opens up brand new possibilities for
  choreographers
• As new creative tools for enhancing their
  performance

10
Major Contributions (4/4)

• Provides a seamless interface to Max/MSP and
  JITTER software packages using a wireless
  interface board
• A choreographer can replace their current
  installation of interactive environment with
  proposed platform
• Without any extra work on the software side

11
Outline

• Abstract
• Introduction
• Related Work
• Platform Design
• Application Example
• Conclusions and Future Work

12
Related Work

• Interactive dance performance
• Use the data from tracking the motion of a dancer
  to steer the generation of musical or graphical
  effects in real time
• Motion tracking technologies
• Computer vision based
• Embedded or wearable sensors based

13
Wearable Wireless Sensing Systems

• Expressive Footware
• A set of piezoelectric acceleration sensors are
  embedded into a pair of dancing shoes
• 19.2Kbps radio

14
Sensor Stack

• Second generation Footware
• 3-axial acceleration sensing
• 3-axial angular velocity measurement
• 115.2Kbps

15
Wireless Inertial Measurement System (WIMS)

• Flexible PCB
• 10?10?10 mm3
• Prone to breaking
• Not suitable for mounting inertial sensors
• Does not include a microcontroller, RF interface,
  and battery

16
Max/MSP and JITTER

• Max/MSP
• A graphical environment for music, audio, and
  multimedia
• Max for MIDI, I/O control, user interface, and
  timing objects
• MSP a set of audio processing objects
• JITTER
• A set of matrix data processing objects optimized
  for video and 3-D graphics

17
Max/MSP and JITTER
18
Outline

• Abstract
• Introduction
• Related Work
• Platform Design
• Application Example
• Conclusions and Future Work

19
Platform Design

• Proposed wearable wireless sensor platform
  consists of three parts
• wearable wireless sensor nodes
• wireless data aggregators
• wireless interface boards

20
Platform Design
21
Wearable Wireless Sensor Nodes

• Activities and physical conditions of a dancer
  are first sensed and digitized by a set of
  wireless sensor nodes that the dancer wears
• Then, each sensor node wirelessly transmits its
  data to the data aggregator worn on the dancers
  waist.

22
Wireless Data Aggregators

• Data aggregator collects and packetizes these
  data and send them to the wireless interface board

23
Wireless Interface Boards

• Converts the received data into digital/analog
  output signals
• feeds them to the MIDI I/O terminal, which
  generates MIDI signals
• Taking these MIDI signals as inputs
• Max, MSP and JITTER process them and synthesize
  musical and visual effects as programmed by a
  choreographer
• Effects are sent to the front projector, speaker,
  and lamps and displayed

24
Scalability Issue

• Uses two different networks
• A body network
• An 802.11b Wi-Fi network
• Similar to Intels heterogeneous network
  architecture
• Enables proposed platform to simultaneously
  monitor the activities and physical conditions of
  multiple dancers
• Without degradation as the number of dancers
  increases

25
Is A Single Network Scalable?

• NO!
• Every sensor node on each sensor would directly
  transmit its sampled data to the interface board
• A body network for each dancer
• A set of wearable wireless sensor nodes
• One data aggregator
• Use 2.4GHz ISM band radio
• Use TDMA-based MAC protocol
• Maximum data rate 250Kbps
• Transmission power level 0dBm

26
802.11b Wi-Fi Network

• Formed by the data aggregators on dancers and
  theater equipment
• Overlaid on the body networks
• Each data aggregator is linked to the access
  point of the interface board using 802.11b CF
  wireless card
• Theoretically, up to 256 data aggregators can be
  connected to the access point simultaneously
• In practice, 10-16 is more like the proper number
  of data aggregators connected to one access point
• To guarantee the required bandwidth

27
Wearable Wireless Sensor Node

• Built based on the design of Eco
• Three variants of Eco
• Wireless transmitter unit (WT)
• Acceleration, temperature, and light sensing unit
  (ATLS)
• Image and gyro sensing unit (IGS)

28
Wireless transmitter unit (WT)

• Includes only a microcontroller and radio
  interface
• With digital input/output and analog input
  interfaces
• To connect to some big sensors
• joint angle sensor
• heartbeat sensor
• infrared sensor
• Turn on/off lanterns

29
Wireless transmitter unit (WT)

• nRF24E1
• A 2.4GHz transceiver with an embedded
  8051-compatible microcontroller (DW8051)
• DW8051 has a 512Byte ROM, a 4KByte RAM, one SPI
  (3-wire) interface, and a 9-channel 12-bit AD
  converter
• Transceiver uses a GFSK modulation scheme in the
  2.4GHz ISM band
• 125 different frequency channels that are 1MHz
  apart
• A chip antenna
• 32K EEPROM
• Maximum RF output power 0dBm
• Maximum data rate 1Mbps
• Maximum power consumption 28mA at 3V

30
Acceleration, temperature, and light sensing unit
(ATLS)

• Consists of
• one 3-axial accelerometer (H34C)
• A 3-axial accelerometer from Hitachi Metals
• Acceleration measurement range is 3g
• It measures only 3.4 3.7 0.92 mm3 and
  consumes 0.36mA at 3V
• one temperature sensor (embedded on H34C)
• -20?C to 65?C
• one light sensor (CdS photoresistor)
• as well as what the WT unit has

31
Acceleration, temperature, and light sensing unit
(ATLS)
32
Image and gyro sensing unit (IGS)

• Built based on the Eco-Stick
• a variant of the Eco
• microcontroller and radio interface
• has either an image sensor (VS6650) or a
  gyroscope (ADRS150)
• VS6650
• 1.0-megapixel SMIA Camera Module from
  STMicroelectronics
• measures 9.5 9.5 7.6 mm3
• consumes 30mA at 3V
• interfaces with the nRF24E1 chip via the SPI port

33
Image and gyro sensing unit (IGS)

• ADXRS150
• a gyroscope from Analog Devices
• measurement range is -150 to 150 degrees
• current consumption of 13mA at 5V
• measures 7 7 3.2 mm3
• Equipped with TWO 40mAh Li-Polymer batteries

34
Wireless Data Aggregator

• Consists of
• An MSP430 16-bit microcontroller
• An nRF2401A 2.4GHz transceiver
• A WCF12 CompactFlash 802.11b card
• 802.11b wireless card consumes a maximum of 250mA
  at 3.3V
• Use a 700mAh Li-Polymer battery
• To guarantee a minimum lifetime of one hour

35
Wireless Data Aggregator
36
Interface Board

• Consists of
• An MC9S12NE64 16-bit microcontroller with a
  built-in fast Ethernet control
• One RJ-45 connector
• Two serial ports
• Digital/analog signal I/O interfaces
• Provides a seamless interface between proposed
  platform and Max/MSP/JITTER software
• Receives TCP/IP packets from data aggregators
• Outputs digital/analog output signals fed to MIDI
  I/O board
• When Max/MSP/JITTER output signals
• It takes these signals and makes a TCP/IP packet
  that contains a proper destination address and
  control message
• Sends out packets to the data aggregators

37
Interface Board
38
Outline

• Abstract
• Introduction
• Related Work
• Platform Design
• Application Example
• Conclusions and Future Work

39
Application Example

• Dreams in the Forbidden City
• Devised by Yicun Sun
• A live dance performance in an interactive
  environment
• Describes the dreams of five concubines of the
  emperor in the Forbidden City
• Their dreams are to please the emperor

40
Dreams in the Forbidden City
41
Dreams in the Forbidden City

• According to the five dancers movement, the
  expression of the emperor varies
• Sometimes the emperor punishes his concubines by
  thunder and lightning
• Other times he expresses cheers by sweet music
  and bright light
• All kinds of sound and visual effects are
  generated by a computer without any manual control

42
Conventional Interactive Environment
43
Improvements by Our Platform

• Eliminate wiring between the stage equipment and
  the Max I/O terminal

44
Improvements by Our Platform
45
Outline

• Abstract
• Introduction
• Related Work
• Platform Design
• Application Example
• Conclusions and Future Work

46
Conclusions

• Propose a wearable wireless sensor platform for
  an interactive dance performance
• Consists of
• wearable wireless sensor nodes
• data aggregators
• wireless interface boards
• Distinguishing features
• Truly wearable
• Highly scalable
• Multi-modal sensing
• Seamless interface

47
Future Work

• Choreographing interactive dance performances
  that take full advantage of proposed platform
• At least tens of dancers will be performing
  together
• Images transmitted from the dancers ISG units
  will be used to synthesize graphical effects on
  the stage
• Dancers heartbeats and body temperatures will be
  monitored and converted into different colors and
  beats to reflect the dancers conditions on the
  stage equipment

48
My Comments

• Strength
• Very good hardware technology
• Interesting and practical applications
• Cooperation with other departments in NTU?
• Weakness
• Truly scalable?
• Interference between wireless sensor nodes of
  different dancers when they get close?

49
Questions?
50
Thank you very much for your attention!
51
Eco

• The sensor node is called Eco
• A world record setter for being the smallest
  wireless sensor node to date.
• Only 557mm3 in volume and 1.6 grams in weight
• Designed to be worn on the limbs of pre-term
  infants to monitor their spontaneous movement in
  response to assisted exercises
• Only 11 the volume of the smallest of the most
  popular commercial sensor node, the Mica2DOT from
  Crossbow.
• Low Power Design Contest Award (another 2,825 in
  cash prize) at ISLPED 2004.

52
Heterogeneous Sensor Networks

• Ad hoc sensor network with a high bandwidth
  802.11 mesh overlay network based on Intel
  XScale technology

53
PXA250 (Intel XScale core) Sensor Gateway
54
Path across overlay network