Action line : Networked Audiovisual Systems and Home Platforms Project N507913 SemanticHIFI Progress

1
Action line Networked Audiovisual Systems and
Home PlatformsProject N507913 -
SemanticHIFIProgress Meeting, Sept. 05 Final
Prototypes, WP 5 Sergi Jorda, Jordi Janer, Alex
LoscosUPF
2
WP5 PerformingAchieved work

• M1 - Deliverable D1.2.1 Functional specification
  and refined workplan use case and functional
  module definition
• M2 - Deliverable D5.1 First Functional Demos
• Conducting tempo change by audio tapping (with
  swing transformation)
• Conducting tempo change with HIFI comandment
• Karaoke voice transformations
• Voice control

3
WP5 Performing Achieved work

• In M4 (Deliverable D5.2 First performing
  functional prototypes ) we started the migration
  towards a VST plugin architecture (development of
  6 independent plugins)
• Conductor
• Synchronizer
• Karaoke transformer
• Voice instrumentizer
• BeatBoxer
• DJ Voice-controller

4
WP5 Performing M6 (the HIFI System)

• Only a very simplified version of the Conductor
  is planned to be included in the HIFI System
  (D1.2.3) ? Time stretch (by means of a speed
  button, ranging from 0.5 to 2.0)
• Input data
• Original audio stream
• Data extracted from the pre-analysis process
• Output data
• Time-modified audio stream
• APIs Interface functions of the LADSPA
  architecture
• Operative details
• Platform and development tools C, Linux Ubuntu
  distribution, gcc compiler.
• Development technologies LADSPA
• External Libraries FFTW, LibAudioFile

5
WP5 Performing M6 (the Authoring System)

• Singalong with voice transformations (Voice
  Transformer VST)
• Voice controlled instruments (VoCtro, BillaBoop
  Wahwactor VSTs)
• Swing transforming (Groovator VST)

6
WP5 Performing Voice Transformer

• Several singing voice transformations are
  included in this real-time VST plugin. The system
  uses spectral techniques for modifying the
  character of the voice. Transformations include
  transposition, vibrato, breathiness, timbre
  mapping and other high-level operations.
• input data audio stream
• output data audio stream
• Uses the standard API of the VST-SDK 2.3.
• GUI consisting of various sliders control
  different transformation parameters a set of
  high-level presets

• Operative details
• Platform and development tools Windows XP, MS
  Visual NET
• Development technologies C, VST SDK 2.3.
• External Libraries FFTW2.1, Libsndfile,
  WinPthreads

7
WP5 Performing Voice controlled instruments

• Digital Musical Synthesis Processing Human
  Voice Analysis ?
• New ways of controlling digital musical
  instruments
• Voice Intrumentizer (VoCTro)
• This plugin synthesizes a trumpet and virtual
  bass guitar that are driven by the input voice in
  real-time. Here the goal is to explore the voice
  as a valid musical controller. Inspired by
  morphing algorithms, were a sound A (voice) is
  morphed into another sound B (instrument), our
  system controls the sound synthesis with a
  microphone signal.
• WahWactor
• Two-input and one-output VST plug-in. Out of the
  two input tracks, one of the tracks may be
  considered a control rather than a proper input
  since it is the one in charge of driving the
  transformations to be applied to the audio
  signal. In the context of the Wahwactor, the
  audio signal is typically a guitar signal and the
  control signal is a voice wa-wa utterance
  signal.
• BeatBoxer (BillaBoop)
• The user sings a rhythm on top of selected
  musical piece. The input voice, which mimics the
  sounds of a drum set, is then analyzed. Timbrical
  aspects of the voice determine which real drum
  set sound is reproduced. In the current version
  we distinguish among three drumset sounds bass
  drum, snare drum and cymbals. Here, the latency
  is a crucial issue, since drumset sounds, are
  particularly sensitive to time deviations.

8
WP5 PerformingVoice controlled instruments
Voice Intrumentizer (VoCTro)

• Control parameters
• Instrument (Trumpet / Elec. Bass), Transposition
  factor (0.5 to 2.0), envelope control (0 to 1.0),
  latency (0 to 10 frames) and output level (0 to
  1.0).
• Input data Audio stream
• Output data Audio stream
• Operative details
• Platform and development tools Windows XP, MS
  Visual NET
• Development technologies C, VST SDK 2.3.
• External Libraries FFTW2.1, Libsndfile,
  WinPthreads

9
WP5 PerformingVoice controlled instruments
WahWactor

• Input data
• Control (e.g. voice) audio stream
• Audio stream to process (e.g. synth, guitar)
• Output data Processed audio stream
• Operative details
• Platform and development tools Windows XP, MS
  Visual NET
• Development technologies C, VST SDK 2.3.
• External Libraries FFTW2.1, Libsndfile,
  WinPthreads

10
WP5 PerformingVoice controlled instruments
BeatBoxer (BillaBoop)

• Input data Control voice audio stream
• Output data MIDI notes (bass drum, snare drum,
  cymbal)
• Operative details
• Platform and development tools Windows XP, MS
  Visual NET
• Development technologies C, VST SDK 2.3.
• External Libraries FFTW2.1, Libsndfile,
  WinPthreads.

11
WP5 PerformingSwing transforming Groovator VST

• This plugin combines Automatic Rhythm Analysis
  and Time-scaling in order to transform the rhythm
  of an audio mix. The whole audio file is first
  pre-analyzed in order to extract a musical
  description. After this process, the user can add
  or substract swing by means of a single slider
  during the playback.
• Control parameters Single slider controls the
  Swing factor during the playback.
• Input data Audio stream
• Output data Audio stream
• Platform and development tools Windows XP, MS
  Visual NET
• Development technologies C, VST SDK 2.3.
• External Libraries FFTW2.1, Libsndfile,
  WinPthreads

12
WP5 PerformingAdd. Musical interaction with
handheld devices
13
(No Transcript)
14
WP5 Performing Objectives for next yearly
period

• M5 - Deliverable D1.2.3 (September 2005) Final
  specifications
• Final specifications of music sharing facilities
  (i.e. what will  home performers  be able to do
  with their mixes and versions?)
• Final specifications of music-analysis metadata,
  to be used by the performing modules in a
   semantic , context-dependent environment
• M6 - Final functional prototypes with
  documentation
• Finish and document the prototypes described in
  M4
• Proceed the migration towards a VST plugin
  architecture
• Start the integration within NI environment
• Start applying the specifications described in
  M5, for its use within the HIFI system