The TaiChi System Environment

1
The TaiChi System Environment

• Antonio Camurri and Gualtiero Volpe (DIST), Alain
  Crevoisier (CeTT), Augusto Sarti (PoliMi), Ming
  Yang (UWC)

2
Objective and deadlines

• Development of an integrated platform of h/w and
  s/w modules and tools for TAIs, consisting of
• A kernel based on the new EyesWeb platform
  version 4.0.2 (released at the end of Year 2).
• Hardware devices (SenseWeb) communicating with
  the EyesWeb platform and providing streamed
  signals or processed data to EyesWeb.
• Software library implemented into EyesWeb and
  SenseWeb
• Deadlines/milestones (according to the TA)
• Month 30 D5.2-Initial version of TAI systems and
  platforms
• Month 36 D5.3-Fully integrated TAI systems and
  platforms

3
Main achievements in Year 2

• During Year 2 the work mainly focused on
• The development of new features in the EyesWeb
  kernel according to feedback and needs emerged
  from the preliminary TAI prototypes developed in
  Year 1 and from the evaluation tasks on public
  events.
• The development of new software modules for basic
  processing of TAI inputs, e.g., the new module
  for continuous acoustic tracking (PoliMi), the
  integrated discrete and continuous tracking for
  La Biennale di Venezia (DIST), discrete tracking
  (CeTT)
• The integration of new software modules for
  high-level gesture analysis and multimodal
  feature extraction from TAI (modules in WP 4 have
  been integrated in EyesWeb).
• The communication layer between EyesWeb and
  SenseWeb.
• Sensors acquisition system has been realized for
  SenseWeb.

4
The TaiChi System Environment
High-level gesture analysis libraries (DIST)
Basic TAI libraries (PoliMi, LOA, UWC, CeTT,
DIST, BU, TUC)
Kernel the EyesWeb platform (DIST)
Prototype user interfaces (CeTT)
SenseWeb DSP module (CeTT)
TAI Sensors (UWC)
5
Sensor technologies

• Our experiments on sensors were based on two
  basic techniques developed for in-solid acoustic
  source localizations Time Delay Of Arrival
  (TDOA) and Location Template Matching (LTM). The
  main work for in-solid acoustic sensing
  technology focused on the following four areas
• test and characterization of various acoustic
  sensors available on market,
• development of piezo sensors for specified
  applications,
• development of specific electronics for signal
  collecting and conditioning,
• development of software platform for sensor
  characterization.
• The main objective of our experiments on acoustic
  sensors and electronics is to determine possible
  variables affecting system results and then to
  devise measures to eliminate negative effects.
  Some possible variables are listed
• Sensor type (e.g. microphone, piezoelectric disk
  and accelerometer)
• Position of sensor (incl. hit locations object
  geometry)
• Object material (e.g. metal, glass, wood board)
• Source or hit type and consistency of hits (e.g.
  finger tap nail click)

6
The electronic circuits developed (UWC)
Two-channel USB soundcard
Digitally-controlled 4-channel amplification board
Amplifiers with dynamic gain control
7
Experiments on sensors

• The experiments we conducted aiming at
  investigating
• what is the best resolution that can be achieved
  with Time Reversal method on different type of
  sensors,
• which type of sensor has the best consistent
  output,
• position and boundary influence on the
  performance of tested sensors,
• Material influence on the performance of tested
  sensors in terms of frequency response and
  sensitivity

8

• Less consistent response from microphones and
  piezo disks for tapping on glass surface

• Consistent responses from microphones and piezo
  disks for tapping on metal sheet

9

• Results from experiments on sensors
• The experimental results showed that there were
  no significant differences in sensor performance
  as a function of its location with respect to
  tangible object boundaries.
• The experimental results also showed that there
  were no significant differences in sensor
  performance in terms of sensitivity, frequency
  responses on most of the tested materials.
• In order to receive a long term quality input
  signal (repeatable and reliable) sensor
  installation is important, particular for LTM
  method.
• Cable shielding is crucial for noise cancellation
  (particular for those input circuits with high
  impedance) for most piezo sensors. Correct
  connection to accelerometers is vital for
  achieving a high intensity input signal.
• On glass the triggering threshold for microphones
  is low making it susceptible to false triggers.
• Nail tapping on glass showed more localization
  consistency than soft tapping. This may attribute
  to the high frequecy components created and
  relatively high signal intensity caused by nail
  tapping.

10
The sensors tested and developed
Accelerometer
11
Specification comparison of sensors
12
(No Transcript)
13
The TaiChi System Environment
High-level gesture analysis libraries (DIST)
Basic TAI libraries (PoliMi, LOA, UWC, CeTT,
DIST, BU, TUC)
Kernel the EyesWeb platform (DIST)
Prototype user interfaces (CeTT)
SenseWeb DSP module (CeTT)
TAI Sensors (UWC)
14
The EyesWeb Kernel (1/3)

• The software kernel has been completely
  re-designed and re-implemented in Year 1 (version
  4) with respect to versions available before the
  TaiChi project (version 3).
• In order to meet the new requirements emerged
  from demos, experiments, and public events (e.g.,
  the Summer School at month 18 and other public
  events, in particular at La Biennale Venezia)
  further features and improvements have been
  introduced in Year 2.

15
The EyesWeb Kernel (2/3)

• Major new features introduced in Year 2 include
• Introduction of novel activation mechanisms of
  EyesWeb modules a single input multiple output
  mechanism enables, e.g., the extraction of
  multiple values from a single audio buffer
  (needed for analysis of audio signals from TAIs)
• Improved synchronisation between
• (i) different audio channels at high and
  different sampling rates, and
• (ii) audio streams at different sampling rates
  and other modalities/channels (e.g. Video) for
  real-time integrated processing.
• Synchronized processing of input channels at
  different sampling rates re-buffering,
  resampling, interpolation, time stretching,
  adaptation to simultaneous synchronised
  processing of different streams, data conversion,
  support to user-defined plugins (e.g. on time
  stretching to maintain synchronisation between
  channels), etc.

16
The EyesWeb Kernel (3/3)

• ...Major new features introduced in Year 2
• Extension of datatypes for representation and
  processing of synchronized TAI information e.g.,
  container datatypes specifically designed for
  handling synchronized multimodal information
  (e.g., audio and video features related to the
  same gesture on a TAI)
• Improved SDK with a particular focus on support
  for easy porting of TAI modules into EyesWeb as
  part of the TaiChi System Environment.
• Introduction of profiling tools, especially
  useful for evaluating the computational load of
  the different processing phases of TAI analysis,
  e.g., audio analysis, video analysis, multimodal
  integration,

17
The TaiChi System Environment
High-level gesture analysis libraries (DIST)
Basic TAI libraries (PoliMi, LOA, UWC, CeTT,
DIST, BU, TUC)
Kernel the EyesWeb platform (DIST)
Prototype user interfaces (CeTT)
SenseWeb DSP module (CeTT)
TAI Sensors (UWC)
18
SenseWeb (CeTT)

• A modular hardware platform for the development
  and exploitation of standalone multimodal
  applications

Data acquisition card(s)
The architecture
Processing card
Communication card
19
SenseWeb - Example
20
SenseWeb - Modules
Miscellaneous Outputs (Relays), Accessories
(battery pack, bred board)
21
SenseWeb - Progress

• March 2005 Partial SenseWeb working
• processing / communication hardware
• TDOA software with simulated input
• TDOA not fully tested, basic communication
• March 2006 Demo. of SenseWeb working
• sensors amplification acquisition
  processing communication hardw.
• acquisition / improved TDOA software
• problem new requirement coming from latest
  TAICHI developments real-time adptative gain
  control, sensor's resonance filtering
• problem acquisition board bug with the trigger
  1.5s reset time

22
The TaiChi System Environment
High-level gesture analysis libraries (DIST)
Basic TAI libraries (PoliMi, LOA, UWC, CeTT,
DIST, BU, TUC)
Kernel the EyesWeb platform (DIST)
Prototype user interfaces (CeTT)
SenseWeb DSP module (CeTT)
TAI Sensors (UWC)
23
Basic TAI libraries (1/2)

• Hybrid TDOA-TR localization (DIST), EyesWeb
• Extraction of features from time and frequency
  representation of TAI audio signals (DIST),
  EyesWeb
• Porting of Sensitive Object module (DIST),
  EyesWeb
• Inverse method for localization of impulsive
  interaction (PoliMi), EyesWeb
• Continuous gesture tracking (PoliMi), EyesWeb
• Active and passive autocalibration procedures for
  the estimation of the elastic parameters of
  materials (PoliMi, LOA), MatLab, C

24
Basic TAI libraries (2/2)

• In-Air holography (BU), Matlab, EyesWeb
• In-Air TDOA (BU), Matlab, C
• SRP-PHAT (BU), Matlab, C
• Active Doppler tracking(BU), Matlab
• TR technique (LOA), Matlab, C
• TDOA techniques (CeTT), DSP and EyesWeb
• In-solid holography (TUC), Matlab

25
Low-Level Gesture Classification

• Design and development of software modules for
  low-level interaction classification how the
  touching occurred (soft finger, nail, friction,
  ...), PoliMi
• Estimating spectral similarity of TAI audio
  signals (for La Biennale di Venezia) (DIST)

26
High-level gesture processing (1/2)

• Features from gesture trajectories. Trajectories
  are extracted by modules developed in Year 1 and
  further enhanced in Year 2. Features include
  kinematical features as well as expressive
  features (e.g., directness, impulsiveness).
• Multimodal integration of sonic and visual cues
  for high-level gesture analysis e.g.,
  impulsiveness estimated from the integration of
  visual (e.g., velocity of the hand approaching
  the TAI) and acoustic information (e.g., dynamic
  range of sound).

27
High-level gesture processing (2/2)

• Multimodal analysis of high-level patterns. E.g.,
  modules for measuring the occupation and the
  possible repetition of gestures on specific areas
  of a TAI, analysis thru gesture spaces. Used for
  (i) high-level gesture interpretation (ii) in the
  evaluation task.
• Machine learning EyesWeb library for classifying
  gestures according to expressive qualities.
  (extension of SVMs, inclusion of SOMs, clustering
  techniques, decision trees).

28
Libraries Development Process (1/3)

• Definition of a protocol to document and porting
  available TAI results in the TaiChi System
  Environment.
• The protocol provides a template to specify and
  document the taxonomy of modules and of research
  prototypes from TAI-CHI.
• The protocol has been defined in Year 2 and it is
  available to the TaiChi partners.

29
Libraries Development Process (2/3)

• Documentation of the research outputs following
  the protocol, i.e., detailing the main features,
  results from testing, assessment evaluation,
  experiments.
• Research outputs include techniques and
  algorithms on TAIs and multimodal gesture
  processing in terms of MatLab, portable C
  classes, or EyesWeb modules.

30
Libraries Development Process (3/3)

• Consolidated research outputs (from WP2 to WP4)
  are ported as real-time modules in the TaiChi
  System Enviroment with the following procedure
• Porting from MatLab to C portable classes
  (engineering and optimisation phase) 
• Inclusion of the C portable class(es) into
  EyesWeb new modules (plugins)
• Possible porting on DSP (a hardware module
  corresponding to the EyesWeb module, when
  appropriate)
• Testing on prototypes (WP5)

31
Prototype user interfaces

• Support for custom user interfaces development
• The new Kernel of EyesWeb is separate from the
  GUI it can run as a background process
• The Kernel can be controlled from the external,
  and has been implemented in portable C. In
  perspective it can run on mobiles

32
IPR issues

• Each external library remains of property of the
  author partner, with separate licensing and
  credits according to the author partner.
• External libraries do not become part of EyesWeb
  they are external plugins, and they become part
  of the TaiChi System Environment.