Title: Distributed%20Multimodal%20Synchronization%20Protocol%20(DMSP)
1Distributed Multimodal Synchronization Protocol
(DMSP)
- Chris Cross
- IETF 65
- March 21, 2006
With Contribution from Gerald McCobb and Les
Wilson
2DMSP Background
- Result of 4 year IBM and Motorola RD effort
- ID to IETF July 8, 2005 by IBM Motorola
- Reason for contribution
- A standard is needed to synchronize network based
services implementing distributed modalities in
multimodal applications - Other protocols may have overlap but do not
address all multimodal interaction requirements - Other IETF IDs and RFCs
- Media Server Control Protocol (MSCP)
- LRDP The Lightweight Remote Display Protocol
(Remote UI BoF) - Media Resource Control Protocol Version 2
(MRCPv2) - Widex
- RFC 1056 Distributed Mail System for Personal
Computers (also DMSP ?)
3Why do you need a distributed system, i.e., a
Thin Client?
A thick client has speech recognition and
synthesis on the device. As resources available
on a device shrink or the application
requirements increase (larger application
grammars) then the performance of the system
becomes unacceptable. When that threshold is
reached then it is economically feasible to
distribute the speech over the network.
Grammar Size and Complexity G
R Resources memory and MIPS on the client
device G Size and Complexity of application
grammars R/G 1 Resources are adequate to
perform real time recognition and synthesis.
4DMSP Architecture
- There are 4 DMSP building blocks
- Modalities
- Model-View-Controller (MVC) design pattern
- View Independent Model
- Event-based modality synchronization
5DMSP Architecture Building Blocks
- Modalities are Views in the MVC Pattern
- GUI, Speech, Pen
- Individual browsers for each modality
- Compound browsers for multiple modalities
Compound Browser
6DMSP Architecture Building Blocks
- Model-View-Controller (MVC) design pattern
- Multimodal system can be modeled in terms of the
MVC pattern - Each modality can be decomposed and implemented
in its own MVC pattern - A modality can implement a view independent model
and controller locally or use one in the network
(e.g., an IM)
7DMSP Architecture Building Blocks
- View Independent Model
- Enables a centralized model
- Modality interaction updates view and model
- Local event filters reflect important events to
view independent model - A modality listens to view independent model for
only the events it cares about - Compound clients, centralized control
(Interaction Manager) as well as distributed
modalities all enabled with a single protocol
8DMSP Architecture Building Blocks
- Event-based synchronization
- Compound Client All modalities rendered in
client - Interactions in one modality reflected in others
thru event based changes to one or more model - GUI DOM serves as View Independent model
9DMSP Architecture Building Blocks
- Event-based synchronization (CONTD)
- Distributed Modality A modality is handled in
the infrastructure - Requires the DMSP for distributing modality
- Event based synchronization via View Independent
Model gives a modality independent distribution
mechanism - Enables multiple topographies
- Compound Client w/ Distributed Modality
- Interaction Manager
10DMSP Design
- There are 4 abstract interfaces
- Command
- Response
- Event
- Signal
- Each interface defines a set of methods and
related data structures exchanged between user
agents - Specified as a set of messages
- XML and Binary message encodings
11DMSP Message Types
- Signals
- One-way asynchronous messages used to negotiate
internal processing states - Initialization (SIG_INIT)
- VXML Start (SIG_VXML_START)
- Close (SIG_CLOSE)
12DMSP Message Types
- Command and control messages
- Add and remove event listener (CMD_ADD/REMOVE_EVT_
LISTENER) - Can dispatch (CMD_CAN_DISPATCH)
- Dispatch event (CMD_DISPATCH_EVT)
- Load URL (CMD_LOAD_URL)
- Load Source (CMD_LOAD_SRC)
- Get and Set Focus (CMD_GET/SET_FOCUS)
- Get and Set Fields (CMD_GET/SET_FIELDS)
- Cancel (CMD_CANCEL)
- Execute Form (CMD_EXEC_FORM)
- Get and Set Cookies (CMD_GET/SET_COOKIES)
13DMSP Message Types
- Responses
- Response messages to commands
- OK (RESP_OK)
- Boolean (RESP_BOOL)
- String (RESP_STRING)
- Fields (RESP_FIELDS)
- Contains 1 or more Field data structures
- Error (RESP_ERROR)
14DMSP Message Types
- Events
- Asynchronous notifications between user agents
with a common data structure - Events correlated with event listeners
- DOM events
- DOMActivate, DOMFocusIn, and DOMFocusOut
- HTML 4 events
- Click, Mouse, Key, submit, reset, etc
- Error and abort
- VXML Done (e.g., VoiceXML form complete)
15DMSP Message Types
- Events (CONTD)
- Form Data
- One or more Field data structures (GUI or Voice)
- Recognition Results
- One or more Result data structures with raw
utterance, score, and one or more Field data
structures - Recognition Results EX
- One or more Result EX data structures with raw
utterance, score, grammar, and semantics - Start and stop play back
- Play back of audio or TTS prompts has started or
stopped - Start and stop play back mark
- TTS encounters a mark in the play text
- Custom (i.e., application-defined)
16DMSP Conclusion
- A protocol dedicated to distributed multimodal
interaction - Based on the Model-View-Controller design pattern
- Enables both Interaction Manager and Client based
View Independent Model topographies - Asynchronous signals and events
- Command-response messages
- Can be generalized for other modalities besides
GUI and Voice - Supports application specific result protocols
(e.g. EMMA) through extension TBD - Interested in getting more participation