Example application: Complete Melody Likelihood for Query By Melody

1
Example applicationComplete Melody
Likelihoodfor Query By Melody
Dynamic Sound Descriptions Matching Between
Sound and Higher Level Features CUIDADO WP2.1.7

• Shai Shalev and Shlomo Dubnov

2
Possible Applications

• Sound Palette
• Single note micro segmentation (ASR).
• Classification using HMM / dynamic features.
• Modeling drum loops and melody contours.
• Music Browser
• - Complete melody likelihood for Query by melody.
• In general any sequence of local features that
  depends on higher level information (likelihood
  measure for the local feature given the higher
  feature and likelihood measure for the time
  evolution (duration, transition prob.)).

3
Challenge

• Matching between a high feature (score) and a
  low feature (sound).
• Modeling dynamic features (Tempo).
• Explicit duration model
• Estimating local features (pitch) in a noisy
  environment (polyphonic music).

4
Solution Ideas

• A statistical framework.
• Using Bayesian Network for matching between
  features at different levels.
• Using latent segmentation variables for
  separating between different levels.
• Modeling the dynamics of tempo using Markov
  assumptions.
• A likelihood measure for pitch (Maximum
  Likelihood).
• A global optimal solution using a Viterbi-like
  algorithm.

5
Complete Melody Likelihood for Query by Melody
?

6
Complete Melody Likelihood for Query by Melody


7
Related Works

• C. Raphael. Automatic segmentation of acoustic
  musical signals using hidden markov models.
• N. Orio and D. Schwartz. Alignment of Monophonic
  and Polyphonic Music to a Score.
• A. S. Durey and M. A. Clements. Melody spotting
  using hidden Markov models.

8
A Statistical Framework
P(Sound Score , Perlman)
P(Sound Score , Menuhin)
9
A Statistical Framework

• A performance consists of
• Score (e.g. the durations and pitches of notes)
• Instrument (e.g. voice, violin, piano)
• Tempo (acceleranado, rallentando etc.)
• Dynamics (forte, piano)
• Articulation/Expression (falcetto, sotto voce,
  )
• Accompaniment (orchestra, piano )

10
A Statistical Framework
Bayesian
Maximum Likelihood
11
A Bayesian Network
Score
Tempo
Vibrato
Accompaniment
Timbre
Actual Duration
Actual Pitch
Sound
12
A Bayesian Network
Score
Tempo
P(Tempo)
Aligned Score
Aligned Score f(Tempo, Score)
Sound
P(Sound Aligned Score)
13
Tempo Probability

• A high level feature.
• A dynamic feature.
• Modeling as a first order Markov sequence.

14
Likelihood Measure for Pitch

• A harmonic model

Leads to
15
Likelihood Measure for Pitch

• A spectral model

Leads to
16
Global Optimal Solution

• Dynamic programming (Viterbi-like).

17
Results

• 32 melodies from famous opera arias (extracted
  from Midi files).
• 93 performances of opera arias performed by 44
  tenors with full orchestral accompaniment.
• Total playing time 6.5 hours.

18
Results
19
Results

• Average Precision 0.90
• (av. num. of hits in the rank list)
• Coverage 1.82
• (av. Index in the ranked list to find all
  hits)
• One Error 0.18
• (Prob. to fail in the first position)

20
Future Work

• Single note micro Segmentation need a labeled
  database or likelihood measure for local features
  given the different parts of the note.
• Learning procedures in Bayesian Network - EM.
• Coordination with other partners testing
  similar methods on additional dynamic/high level
  features (drum loops? Warm sound?).