Music Representation, Searching, and Retrieval a.k.a. Organization of and Searching in Musical Infor - PowerPoint PPT Presentation


PPT – Music Representation, Searching, and Retrieval a.k.a. Organization of and Searching in Musical Infor PowerPoint presentation | free to download - id: 2f231-ZDMxN


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation

Music Representation, Searching, and Retrieval a.k.a. Organization of and Searching in Musical Infor


School of Informatics & School of Music. Indiana ... Pitches (how high or low): on vertical axis. NP2. ... 3. Why is Musical Information Hard to Handle? ... – PowerPoint PPT presentation

Number of Views:100
Avg rating:5.0/5.0
Slides: 31
Provided by: Donal6


Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: Music Representation, Searching, and Retrieval a.k.a. Organization of and Searching in Musical Infor

Music Representation, Searching, and Retrieval
(a.k.a. Organization of and Searching in Musical
  • Donald Byrd
  • School of Informatics School of Music
  • Indiana University
  • 12 January 2008

  • 1. Introduction and Motivation
  • 2. Basic Representations
  • 3. Why is Musical Information Hard to Handle?
  • 4. Music vs. Text and Other Media
  • 5. OMRAS and Other Projects
  • 6. Summary

Classification Logician Generals Warning
  • Classification is dangerous to your understanding
  • Almost everything in the real world is messy,
  • Absolute correlations between characteristics are
  • Example some mammals lay eggs some are naked
  • Example was the first real piano Cristofori's
    (ca. 1700), Broadwood's (ca. 1790), or another?
  • People say an X has characteristics A, B, C…
  • Usually mean an X has A, usually B, C…
  • Leads to
  • People who know better claiming absolute
  • Is it this or that or that? questions that
    dont have an answer
  • Don changing his mind
  • But lack of classification is dangerous to
  • So should we abandon hierarchic classifications?
  • Of course not they're much too useful
  • Just to be on guard for misleading things in

1. Introduction and Motivation (1)
  • The Fundamental Theorem of Music Informatics
  • Music is created by humans for other humans
  • Humans can bring tremendous amount of contextual
    knowledge to bear
  • In fact, they can't avoid it, and they're rarely
    conscious of it
  • But (as of early 2008) computers can never bring
    much contextual knowledge to bear, often none,
    never without being specifically programmed to
  • gt doing almost anything with music by computers
    is very difficult many problems essentially
  • For the forseeable future, only way to make
    significant progress is by doing as well as
    possible with little context, sidestepping
    intractable problems
  • Not a theorem (I recently made it up), but

1. Introduction and Motivation (2)
  • Three basic forms (representations) of music are
  • Audio most important for most people (general
  • All Music Guide ( has info
    on gtgt230,000 CDs
  • MIDI files often best or essential for some
    musicians, especially for pop, rock, film/TV
  • Hundreds of thousands of MIDI files on the Web
  • CWMN (Conventional Western Music Notation) often
    best or essential for musicians (even amateurs)
    music researchers
  • Music holdings of Library of Congress over 10M
  • Most is notation, especially CWMN, not audio
  • Includes over 6M pieces of sheet music and
    tens/hundreds of thousands of scores of operas,
    symphonies, etc.
  • Differences among the forms are profound
  • NB all statistics above are several years old

2. Basic Representations of Music Audio (1)
Audio (e.g., CD, MP3) like speech
Time-stamped Events (e.g., MIDI file) like
unformatted text
Music Notation like text with complex formatting
Basic Representations of Music Audio (2)
  • Audio Time-stamped Events Music Notation
  • Common examples CD, MP3 file Standard MIDI
    File Sheet music
  • Unit Sample Event Note,
    clef, lyric, etc.
  • Explicit structure none little (partial voicing
    much (complete
  • information) voicing information)
  • Avg. rel. storage 2000 1 10
  • Convert to left - easy OK job easy
  • Convert to right 1 note pretty easy OK job
    fairly hard -
  • other hard or very hard
  • Ideal for music music music
  • bird/animal sounds
  • sound effects

Basic Representations of Music Audio (3)
Audio no (explicit) structure
Events/MIDI simple structure
  • Notation very complex structure

The Four Parameters of Notes
  • Four basic parameters of a definite-pitched
    musical note
  • 1. pitch how high or low the sound is
    perceptual analog of frequency
  • 2. duration how long the note lasts
  • 3. loudness perceptual analog of amplitude
  • 4. timbre or tone quality
  • Above is decreasing order of importance for most
    Western music
  • …and decreasing order of explicitness in CWMN!

How to Read Music Without Really Trying
  • CWMN shows at least six aspects of music
  • NP1. Pitches (how high or low) on vertical axis
  • NP2. Durations (how long) indicated by note/rest
  • NP3. Loudness indicated by signs like p , mf ,
  • NP4. Timbre (tone quality) indicated with words
    like violin, pizzicato, etc.
  • Start times on horizontal axis
  • Voicing mostly indicated by staff in complex
    cases also shown by stem direction, beams, etc.
  • See Essentials of Music Reading musical example
  • At the other extreme, see the Gallery of
    Interesting Music Notation!
  • http//

How People Find Text Information
  • What user wants is almost always concepts…
  • But computer can only recognize words

How Computers Find Text Information
  • Stemming, stopping, query expansion are all
    tricks to increase precision recall (avoid
    false negatives false positives) due to
    synonyms, variant forms of words, etc.

3. Why is Musical Information Hard to Handle?
  • 1. Units of meaning not clear there are
    anyassuming music even has meaning! (all
  • 2. Polyphony parallel independent voices,
    something like characters in a play (all
  • 3. Recognizing notes (audio only)
  • 4. Other reasons
  • Musician-friendly I/O is difficult
  • Diversity of styles of music, of people
    interested in music
  • Music is an art!
  • Cf. Byrd Crawford (2002)
  • But what is the information, in the first place?

Units of Meaning (Problem 1)
  • Handling text information nearly always via words
  • What we want is concepts what we have is words
  • Not clear anything in music is analogous to words
  • No explicit delimiters (like Chinese)
  • Experts dont agree on word boundaries (unlike
  • Music is always art gt meaning much more
  • Are notes like words?
  • No. Relative, not absolute, pitch is important
  • Are pitch intervals like words?
  • No. Theyre too low level more like characters

Units of Meaning (Problem 1)
  • Are pitch intervals like words?
  • No. Theyre too low level more like characters
  • Are pitch-interval sequences like words?
  • In some ways, but
  • Ignores rhythm
  • Ignores relationships between voices (harmony)
  • Probably little correlation with semantics
  • Are chords like words? (Christy Keele)
  • If so, chord progressions may be like sentences
  • In some ways, but ignores melody rhythm, most
    relevant for tonal music, etc.
  • Anyway, in much music, pitch isnt important,
    and/or notes arent important!

Independent Voices in Music (Problem 2)
J.S. Bach St. Anne Fugue, beginning
Independent Voices in Text
  • MARLENE. What I fancy is a rare steak. Gret?
  • ISABELLA. I am of course a member of the / Church
    of England.
  • GRET. Potatoes.
  • MARLENE. I havent been to church for years. / I
    like Christmas carols.
  • ISABELLA. Good works matter more than church
  • --Caryl Churchill Top Girls (1982), Act 1,
    Scene 1

Performance (time goes from left to right)
M What I fancy is a rare steak. Gret? I
havent been... I I am of course a member of
the Church of England. G Potatoes.
Music Notation vs. Audio
  • Relationship between notation and its sound is
    very subtle
  • Not at all one symbol ltgt one symbol
  • Notes w/ornaments (trills, etc.) are one gt many
  • All symbols but notes are one gt zero!
  • Bach F-major Toccata example
  • Style-dependent
  • Swing (jazz), dotting (baroque art music)
  • Improvisation (baroque art music, jazz)
  • Events (20th-century art music)
  • How well-defined is style-dependent
  • Interpretation is difficult even for musicians
  • Can take 50-90 of lesson time for performance

Music Perception and Music IR
  • Salience is affected by texture, loudness, etc.
  • Inner voices in orchestral music rarely salient
  • Streaming effects and cross-voice matching
  • produced by timbre Wessels illusion (Ex. 1, 2)
  • produced by register Telemann example (Ex. 3)
  • Octave identities, timbre and texture
  • Beethoven Hammerklavier Sonata example (Ex.4,
  • Affects pitch-interval matching

4. Music vs. Text and Other Media
  • Explicit Structure Salience
  • least medium most increasers
  • Music audio events notation loud thin texture
  • Text audio (speech) ordinary text with
    markup headlining large,
  • written text bold, etc.
  • Images photo, bitmap PostScript drawing-program br
    ight color
  • file
  • Video videotape MPEG? Premiere file motion, etc.
  • w/o sound
  • Biological DNA sequences, MEDLINE abstracts ??
  • data 3D protein structures

Features of Music Text Analogies
  • Simultaneous independent voices and texture
  • Analogy in text characters in a play
  • Chords within a voice
  • Analogy in text character in a play writing
    something visible to the audience while saying
    different out loud
  • Rhythm
  • Analogy in text rhythm in poetry
  • Notes and intervals
  • Note pitches rarely important
  • Intervals more significant, but still very
  • Analogy in text interval (very roughly!)
    letter, not word

Features of Text Music Analogies
  • Words
  • Analogy in music for practical purposes, none
  • Sentences
  • Analogy in music phrases (but much less
  • Paragraphs
  • Analogy in music sections of a movement (but
    less explicit)
  • Chapters
  • Analogy in music movements

Course Overview
  • II. Organization of Musical Information (music
  • What we want is concepts what we have is words
  • Audio, MIDI, notation
  • III. Finding Musical Information
  • A Similarity Scale for Content-Based Music IR
  • IV. Musical Similarity and Finding Music by
  • V. Finding music via Metadata
  • Digital music libraries (Variations2), iTunes,
  • Music recommender systems

1. Programming in R No Problem!
  • R is very interactive can use as powerful
  • Assignments will be fairly simple (though not for
    MusInfo CompSci grad students -) )
  • Much help available from Don other students
  • Why R?
  • designed for statistics, but thats NOT why!
  • easy to do simple things with it
  • easy to do many fairly complex things, incl.
    graphs handling audio files
  • probably not good for really complex programs
  • free, available for all popular operating
  • very interactive gt easy to experiment
  • has good documentation
  • In use in other Music Informatics classes,
    standardizing is good

1. Rudiments of R
  • Originally for statistics, but useful for far
  • How to get R
  • Web site http//
  • Versions for Linux, Mac OS X, Windows
  • Already on STC Windows machines will be in M373
  • Tutorial
  • http//
  • Can use R interactively as a powerful graphing,
    musicing, etc. calculator
  • …but its not perfect sometimes very cryptic

Music Recommender Systems
  • Work by genre classification and/or collaborative
  • Major interest in recently
  • Best known include
  • Pandora (cf. Music Genome Project)
  • MusicStrands
  • Other interesting sites
  • Hype Machine (for savants?)

Typkes MIR System Survey
  • Rainer Typkes MIR Systems A Survey of Music
    Information Retrieval Systems lists many systems
  • http//
  • Commercial system Shazam
  • Some research systems can be used over the Web,
  • C-Brahms
  • Meldex/Greenstone
  • Mu-seek
  • MusicSurfer
  • Musipedia/Tuneserver/Melodyhound
  • QBH at NYU
  • Themefinder

Machinery to Evaluate Music-IR Research
  • Problem how do we know if one system is really
    better than another, or an earlier version?
  • Solution standardized tasks, databases,
  • In use for speech recognition, text IR, question
    answering, etc.
  • Important example TREC (Text Retrieval
  • For music IR, we now have...
  • IMIRSEL (International Music Information
    Retrieval Systems Evaluation Laboratory) project
  • http//
  • MIREX (Music IR Evaluation eXchange) modeled on
  • 2005 audio only
  • 2006, 2007 audio and symbolic

Collections (a.k.a. Databases) (1 of 2)
  • Collections are improving, but very slowly
  • For research poor to fair
  • Candidate Music IR Test Collections
  • http//
  • Representation CMN vs. CMN (assume Western)
  • For practical use pathetic (symbolic) to good
    (pop audio)
  • Most are commercial, especially audio
  • Very little free/public domain
  • …especially audio! (cf. RWC)
  • IPR issues are a total mess

Collections (a.k.a. Databases) (2 of 2)
  • Why is so little available?
  • Symbolic form no efficient way to enter
  • Solution OMR? AMR? research challenges
  • Music is an art!
  • Cf. Searching CMN slides chicken egg problem
  • IPR issues are a total mess