CASPAR Contemporary Arts Testbed: Preservation of Interactive Multimedia Performances

1
CASPAR Contemporary Arts TestbedPreservation
ofInteractive Multimedia Performances

• Kia Ng and Eleni Mikroyannidi
• ICSRiM - University of Leeds
• caspar_at_icsrim.org.uk www.icsrim.org.uk/caspar

2
Overview

• Context and Motivation
• CASPAR EC IST Project Contemporary Arts Testbed
• Interactive Multimedia Performance (IMP)
• Music via Motion (MvM)
• i-Maestro 3D Augmented Mirror (AMIR) System
• ICSRiM Conducting Interface
• Research and Development
• IMP dataset
• CIDOC-CRM and FRBR ontologies
• Result and Conclusion
• ICSRiM IMP Archival System
• Interface and functionalities
• Scenarios
• Validation

3
Context

• CASPAR EC IST Project
• Build a framework to support end-to-end
  preservation lifecycle for scientific, artistic
  and cultural information
• Contemporary performing arts testbed
• Interactive Multimedia Performance (IMP)
• Interactive Multimedia
• Interactive multimedia performances
• Performance capture/recording and analysis beyond
  normal AV

4
Music via Motion (MvM)

• MvM a framework to integrate interactive
  multimedia, motion sensing, audio synthesis, VR
  and AR technologies
• To explore virtual and augmented instruments
• To provide users/performers with real-time
  control of multimedia events with their physical
  movements
• Application areas include stage performance and
  installation arts

5
IMP Elements and Processes
Example of an IMP production process based on the
Music via Motion (MvM) system, www.kcng.org/mvm
6
i-Maestro 3D Augmented Mirror (AMIR)

• Technology-enhanced learning
• More in-depth recording and understanding
• playing gesture
• Intangible heritage
• Individual performance style
• Data stream include audio, video, 3D motion
  data, sensor data, analysis
• Multimodal feedbacks with graphical visualisation
  and sonification
• Max/MSP jitter, c/c, Lua, SDIF

Excerpt from Bach's Partita No. 3 in E major.
BMW 1006 (Preludio)
7
AMIR (www.i-maestro.org)
8
ICSRiM Conducting Interface

• A system for tracking and analysing a conductors
  hand movements
• Uses multiple wiimotes to capture a conductors
  movements
• These movements are analysed
• This information is then fed back to the user in
  an entertaining yet educational manner

9
System Setup
10
GUI

• 3D environment
• Intuitive UI
• Multiple modes
• Multimodal data capture and analysis

11
Complexity of the IMP Preservation

• IMP rely heavily on digital media
• A recording (e.g. video) of a performance is not
  sufficient for re-performance
• Preserving individual components of a performance
  is clearly a challenge
• However, putting these components together in one
  place does not make a performance
• They need to be assembled in a logical and
  temporal order with their inter-relationships
• Preserving this knowledge through time is even
  more challenging
• Preservation for
• Re-performance at a later time
• Historical study/analysis in performing arts in
  the future
• Better preparation for the future needs

12
IMP Dataset

• An Interactive Multimedia Performance (IMP) work
  typically consists of the following digital
  objects
• Audio
• Video
• 2D/3D motion data
• Sensor data
• Max/MSP patches
• Configuration files
• and others
• For reconstruction of an IMP work
• Representation Information
• Inter-relationship
• Logical
• Temporal

13
CIDOC-CRM for IMPs

• The CIDOC Conceptual Reference Model (CIDOC-CRM)
  has been proposed as a standard ontology for
  enabling interoperability among digital libraries
• It defines a set concepts for physical and
  temporal entities

14
CIDOC-CRM for IMPs (2)

• CIDOC-CRM can be used to describe a performance
• at a high level
• More specialised vocabularies are necessary for
  IMPs
• e.g. how different types of equipment are
  connected together in the performance.
• CIDOC-CRM
• lacks concepts for digital objects
• is designed primarily for documentation of what
  has happened, whereas in digital preservation, it
  is also required to document the reconstruction
  of a past event from preserved components.

15
CIDOC-CRM FRBR Extension

• Two main objectives in extending CIDOC-CRM for
  the preservation of an IMP
• To provide a domain specific vocabulary for
  describing an IMP.
• e.g. details on how the archived performance was
  carried out and how it can be re-performed
• To provide a vocabulary for describing digital
  objects, their interrelationships and operations
  performed on them in the digital preservation
  context
• FRBR (Functional Requirements for Bibliographic
  Records)
• describes a high level conceptual model of
  creative works and how they are represented in
  the real world
• It is used as extension ontology of the CIDOC-CRM
  
• for describing the conception process of the
  performance

16
Representation Information

• The Representation Information for this
  performance is in the form of CIDOC-CRM and FRBR
  instantiation.

16
17
Mapping IMP detailsto CIDOC-CRM and FRBR

• The FRBR concept F52.Performance is used to
  model an IMP.
• The FRBR concept F8.Person represents the
  people involved in the performance, e.g.
  directors, performers, MAX/MSP programmers, etc.
• The digital objects used (software patches, 3D
  models, etc) are instances of the
  E73.Information Object CIDOC-CRM class, whilst
  any physical objects (instruments, computer
  equipment, etc) are instances of the
  E22.Man-Made Object class.
• The E52.Time-Span and E53.Place classes are
  used to represent the time-span and location of
  the performance respectively.

18
Mapping IMP detailsto CIDOC-CRM and FRBR (contd.)
19
ICSRiM IMP Archival System

• The ICSRiM IMP Archival System
• for the preservation of different IMPs
• based on the CASPAR Framework
• Integration of the selected CASPAR components
• Communication with the Repository containing the
  Knowledge base and the DataStore

20
Web Interface of the IMP Archival System
21
Cyclops tool

• A tool to capture appropriate Representation
  Information to enhance Virtualisation and future
  re-use of the digital object.
• Cyclops provides a usable interface to describe
  the Performances in the form of graphs
• The graph in the backend is saved as an RDF
  instantiation of the CIDOC and FRBR ontologies
• Defined terminologies according to various data
  types

22
Cyclops Interface
23
Ingestion and Retrieval of an IMP

• Scenario 1 Ingestion of an IMP
• Description of the performance with the use of
  Cyclops
• Generation of the Representation Information
• Both the metadata and the digital files are
  ingested and stored in the Repository
• Scenario 2 Access of an IMP
• Queries are performed on the metadata
• The related objects are returned to the user
• Instructions extracted from the RepInfo provide
  details on how to assemble the objects

24
Scenario 1 Ingestion of an IMP

• Steps in Screenshots

25
Step 1 Describe the IMP

• Follow the Ingest wizard to complete the
  ingestion

26
Step 1 Describe the IMP (2)

• Create Representation Information with the use of
  Cyclops tool
• It is saved in RDF format. In this way, the graph
  was transformed into an RDF instantiation of the
  CIDOC-CRM and FRBR ontologies.

27
Step 2 RepInfo parsing

• The system detects and parses the RepInfo for
  defined data types

28
Step 3 Addition of the digital files

• Upload the related digital files of the
  performance
• The files are linked to the RepInfo
• The package is stored in the Repository

29
Scenario 2 Retrieval of an IMP

• Steps in Screenshots

30
Step1 Search for the IMP

• The search is done by the name of the performance
  

31
Step1 Search for the IMP (2)

• The system returns a table with the following
  selected information from the RepInfo
• The name of the Performance
• Details on the performance
• The associated with the performance files
• The links to download the files of the
  performance
• Details on the files (e.g. which one is the data
  file, the patch file, how to use them etc.)
• A link to the RDF description.

32
Step 2 Download Associated Performance files

• Select and download the components of the
  performance

33
Step 3 Reconstruct the performance

• The user has to understand the connection of the
  components
• Instructions on the connection is provided in the
  table
• By assembling the components the user is able to
  see the performance

34
Retrieval of an IMP
35
Validation and Feedback

• Participant
• A small set of experts who work with IMP
• Validation
• Ingestion of an IMP work (their own work)
• Retrieval and reconstruction of an IMP they have
  no information about it
• Download and connect the components using the
  instructions
• Fill in a questionnaire with their feedback
• Results
• The system is useful to store and preserve the
  IMPs
• The system provided all the needed instructions
  for the reconstruction of an IMP
• The appropriate RepInfo has been captured
• Some difficulty in using Cyclops (complexity of
  the graphs)
• Demo videos and tutorials helped
• Cyclops graph templates to help to get started

36
CASPAR IMP References

• Selected publications include
• E. Mikroyannidi, B. Ong, D. Giaretta and K. Ng,
  Preservation of Interactive Multimedia
  Performance with the use of Ontology Models, in
  Proc. of the Digital Resources for the Humanities
  Arts (DRHA) Conference, Belfast, September 2009
• K. Ng, E. Mikroyannidi, B. Ong, N. Esposito and
  D. Giaretta, "Interactive Multimedia Systems for
  Technology-Enhanced Learning and Preservation"
  in Proc. Of the 15th International Conference on
  Distributed Multimedia Systems (DMS) 2009, San
  Francisco, September 2009.
• K. Ng, T.V. Pham, B. Ong, A. Mikroyannidis, and
  D. Giaretta, "Preservation of Interactive
  Multimedia Performances", International Journal
  of Metadata, Semantics and Ontologies, 3(3), 28
  February 2009 , pp. 183-196, Inderscience
  Publishers.
• For latest publications on CASPAR IMP
  Preservation, see
• www.icsrim.org.uk/caspar
• CASPAR project website

37
Conclusions

• Preserving knowledge is vital for future
  reconstruction of interactive multimedia
  performances
• Proposed IMP preservation process
• Made use of standard ontology models, CIDOC-CRM
  and FRBR to create Representation Information
• Used to define interrelationships between the
  components
• Creation of a web-based archival system for the
  preservation of IMPs
• Integrated selected components of the CASPAR
  Framework
• Current prototype online
• http//www.icsrim.org.uk/imp-preserve

38
Thank you ?

• Kia Ng and Eleni Mikroyannidi
• ICSRiM University of Leeds
• www.icsrim.org.uk/caspar caspar_at_icsrim.org.uk