KEYed User Interface A Novel Music Composition Tool by

1
KEYed User InterfaceA Novel Music Composition
Toolby

• Farhan Mohamed
• supervised by
• Dr. Sidney Fels

2
Motivation

• While composing with my music studio computer,
  I find it stressful to use my piano keyboard,
  computer keyboard and mouse repetitively,
  especially when I feel creative and work over
  long hours
• - Composer/Producer Kevin Kowal

3
Goal

• The KEYed User Interface project was created to
  bind music performance and editing tasks by using
  the piano keyboard as the sole input device for
  all tasks

Kevins Setup using KEYed User Interface
4
Overview of Presentation

• Current music composition workstation setup
• Alternative approaches and their shortcomings
• Applying findings in Human Computer Interaction
• KEYed User Interface system
• Experiments and results
• Conclusions

5
Music Composition Workstation

• Switching between multiple input devices
• Long device acquisition time
• Not comfortable to use
• Lots of physical space occupied by the input
  devices

6
Alternative Approaches

• Interface separates performance and editing

Meredith Brooks Studio Layout
Mackies Human User Interface

• Interface integrates performance and editing
• - Logic Audio Music Software

7
Separated Vs Integrated Control
Separated Edit and Performance
Integrated Edit and Performance
Space Multiplexed Device Time
Multiplexed Device
8
Time Multiplexed Device
Logic? Audio
Integrated Performance and Edit
Our Focus is here
9
Applied Human Computer Interaction Principles

• Mode Errors and their Prevention
• - Kinesthetic, Confirmatory sounds, Earcons

Human Manual Studies

• Tacit Knowledge and Metaphors

10
Overview of KEYed User Interface
11
HCI Principles Applied to KEYed User Interface
Earcons
Bimanual Action
Single-Point Touchpad
Electronic Piano Keyboard
Space Multiplexed
Mode Error Prevention
Momentary Footswitch
12
Piano Mappings
General Edit Octave (C6-B6) Mapping Example
F6
Undo
Key Edit Octave (C5-B5) Mapping Example
C5
Open/Close key edit window
Note
Arrange Octave (C4-B4) Mapping Example
C2 B2 C3 B3 C4 B4 C5
B5 C6 B6

F4
Move down the track
Transport Octave (C3-B3) Mapping Example

C3
Open/Close transport window
Touch Pad Octave (C2-B2) Mapping Example

E2
Selecting volume
13
KEYed UI Prototype 1

• Prototype I - Steinberg Cubase 5.0 VST 32 on
  Windows 98
• KEYed Pedal for Mode Switching
• Octaves and the Notes are labeled

14
KEYed UI Prototype I Studies

• The interaction methods compared are
• a) Computer Keyboard Piano Keyboard
• b) KEYed User Interface Prototype I
• Testing for Response Time with 6 Composers
• Composing Tasks AB, Repeated Once
• Note Task A and B involve repetitive performing
  and editing, thereby requiring the composers to
  switch back and forth between modes on their own.

15
KEYed UI Prototype I Results
Test 1 Test 2

Mean Time with 95 Confidence error bars

• Learning Time 5 minutes
• Results using paired t-test
• Test 1 - Piano controller faster, significant
  (p
• Test 2 - Piano controller faster, not significant
  (p
• No mode errors detected

16
KEYed UI Prototype II
KEYed Pedal
Single-Point Touchpad

• Prototype II platform
• Steinberg Nuendo 1.5 on Windows 2000
• KEYed Pedal and Sustain Pedal
• A single-point touchpad for 1D and 2D tasks

17
KEYed UI Prototype II Studies

• 1. The interaction methods compared are
• a) Computer Keyboard / Mouse Piano Keyboard
• b) KEYed User Interface II with Audio
• c) KEYed User Interface II without Audio
• Testing for Response Time with 10 Composers
• Simple, Moderate and Complex editing tasks
• Note All tasks involve repetitive composing and
  editing, thereby requiring the composers to
  switch back and forth between modes on their own.
  
• 2. Case study for a one week period with 1
  Composer

18
KEYed UI Prototype II Studies
Experiment Booth
KEYed user interface Layout
19
KEYed UI Prototype II Results
Mean Completion times

• Learning Time 16 minutes
• KEYed Prototype II with Audio vs. Computer
  Keyboard / Mouse Response time faster with the
  KEYed UI with no significance
• KEYed Prototype II without Audio vs. Computer
  Keyboard / Mouse Response time faster with the
  KEYed UI with no significance

20
KEYed UI Prototype II Results

• Case study results
• Positive feedback
• Very natural to use
• Requested more mappings
• Suggested the use of musical chords
• Single-point touchpad too small

21
Contributions

• Piano keyboard can be used as an alternative
  input device in music composition workstations
• A momentary foot pedal like the KEYed foot pedal
  is easy to acquire, and is an ideal mode switch
  for switching between the performing and editing
  modes
• Kinesthetic feedback helps in multi mode
  situations
• Though small, the single-point touchpad is found
  to be natural to use for bimanual tasks

22
Contributions

• KEYed user interface illustrates how an
  appropriate mapping of the layout, feedback, and
  context is important in the design of user
  interfaces
• Results published at CHI2002 and ICMC2002

23
Future Directions

• Overlaying composition functions on other
  controllers
• Exploring musical structures and mappings for
  example, scales and chords
• Providing user customizable mappings

24
Conclusions

• Current approaches to music composition
  workstations design divide the performance and
  editing tasks
• HCI findings can be applied to the current
  problems in music composition workstation design
• Gives rise to faster response time in devices
  with minimal learning
• Ultimately, greater comfort leads to greater
  creative flow and better music

25
Questions
26


KEYed UI System Design


Cubase 5.0 VST Or Nuendo 1.5

Windows Applications
Event Windows Queue

KEYed Messaging System

KEYed Dynamic Link Library (Window Handles)
Serial Port
MIDI OUT MIDI IN MIDI THRU

KEYed Sustain Pedal Pedal
Single-point Touchpad

Figure 3.17 KEYed User Interface Software
Design
27
Measured Mean Task Completion Time vs Predicted
KLM Times
28
Measured Mean Task Completion Time vs Predicted
KLM Times
Where, CK Computer Keyboard KP1 KEYed
Prototype I KP2 KEYed Prototype I M
Mouse KP2A KEYed Prototype II with Audio KP2NA
KEYed Prototype II without Audio TP Touchpad
CK KP1 CK KP1
TASK A TASK B
CK/M KP2A KP2NA TP CK/M KP2A
KP2NA TP CK/M KP2A KP2NA TP
TASK C (Simple)
TASK D (Moderate) TASK E
(Complex)
29

• Single point touchpad
• Ex. Drawing graphs with the touchpad

30
(No Transcript)
31
(No Transcript)
32
(No Transcript)
33
(No Transcript)
34
HCI in Space Multiplexed Device
Continuum Piano Interface (Haken)
Haptic Knob (MacLean, K)