OpenInterface Project FP635182 1 sept 2006 33 months Today M9

1
Welcome
2
OpenInterface Project FP6-351821 sept 2006 - 33
monthsToday M9

• Coordinator Laurence Nigay, University of
  Grenoble 1 - Joseph Fourier
• Glasgow, May 24, 2007

3
Outlines

• Overview
• OI Platform
• Component based platform
• Runtime Kernel
• Repository of modalities
• Graphical editor

4
Overview
5
Consortium
6
Consortium

• Complementary expertise

Signal Processing UCL MULT
Multimodal Interaction ALL
Hardware UGLA SWIN
Software Engineering UJF UGLA FIT ARC PC
LI TXT
Ergonomics Usability FIT FT
Application domain Large information space FT
UJF UGLA
Application domain Game ARC FT UJF UGLA
7
OpenInterface objectives
8
The 2 action lines

• Two complementary goals Extend the
  sensory-motor capabilities of mobile devices
  and environments

Interaction Continuity
9
The 2 action lines

• Two complementary goals Extend the
  sensory-motor capabilities of mobile devices
  and environments

Modalities
10
Why 2 action lines?
Market

Existing standards for multimodality
Standards
Multimodal Interaction on a mobile device
Validation
Multimodal Interaction within a pervasive
environment
RD
Ideas
OpenInterface platform for multimodality
Concept
Concept
11
Why 2 action lines?
Market

VALIDATORS
Standards
Multimodal Interaction on a mobile device
Validation
Multimodal Interaction within a pervasive
environment
TESTBEDS
RD
Ideas
Concept
Concept
12
Testbed

• Large Information Space

13
Testbed

• Game

Speech command for the speed
Accelerometer or speech
Increasing the intensity of the vibration
depending on how damaged the helicopter is
14
Testbeds and validators
TESTBEDS
Ergonomic evaluation
Ergonomic evaluation
Software prototype
Software prototype
Requirements
Design
Design
Ergonomic evaluation
Design
Software prototype
Ergonomic evaluation
Software prototype
OpenInterface Platform
15
OI Platform
16
OI Platform

• Open source platform
• To make it easier to develop effective
  interactive systems that exploit multimodal
  interaction techniques in modern application
  settings (mobile, ubiquitous, context sensitive)

17
OI Platform

• Open source platform for developing multimodal
  interfaces
• that enables quick replication,
• that enables iterative user-centred design,
• that contains a rich and extensible set of
  modalities,
• that enables a focus on innovation (new
  modalities or forms of multimodality),
• that supports dynamic selection and combination
  of modalities to fit the ongoing context of use.

18
OI Platform
- Repository of modalities (components) -
Repository of Fusion/Fission Mechanisms
(components) - Runtime Kernel
19
OI platform

• Who are the users?
• Three types of users
• Programmers
• Providing new components (populating the
  platform)
• New device
• New modality
• New fusion/fission mechanism
• Multimodal Application Designers
• Assembling components to define multimodal
  interaction
• End-users of multimodal applications
• Interacting with the multimodal application
• Adaptation at run-time

20
OI platform Runtime kernel
21
Outlines

• Kernel
• Overview
• OI Components
• CIDL, PDCL
• Current Features
• Next Features

22
Overview

• Useful high level programming language support
• C/C, Java, Matlab, Python

• Plug n Play behavior.
• Lightweight skeleton platform.

23
Components

• Platform does not specify/define a Component
  Model (!!)
• Platform sees the external software as components
• components are assumed to behave like objects.
• Software/Component delivered as reusable
  independent unit with exported/imported I/O
  interfaces.
• Platform Kernel aims for the interaction of
  heterogeneous components

24
Component OI point of view

• Facets can be seen as services provided by a
  component
• A Component is a set of facets
• This is how any external software is seen by the
  Kernel

25
Reusable Independent Unit

• Software
• With API
• With installation/configuration doc
• Without explicit dependencies with other
  components
• Tested and validated
• Much likely a library (!!)
•  Impossible to test new modalities with an exe,
  for instance.

26
Exported/Imported IO

• Components only declare communication interface
• Enforce the requirement for independence
• Export IO (functions, etc.) when providing a
  functionality
• Display an image
• Import IO for requesting a functionality
• An event handler

27
OpenInterface CIDL

• Designed to describe Independent and Reusable
  component.
• (Intentionally) Not as expressive as WSDL (etc.)
• Restricted on purpose to cover common attributes
  of various programming languages
• Describes component interface using a programming
  language independent description (XML).
• Not intended to be written by human
• Generation tools are not yet available though
  (being developed)

28
I/O Interface

• Functions provided by a component facet
• 3 types of functions
• Sink
• With no return argument
• Source
• With return argument
• Callback
• With no return argument, for importing external
  functions

29
I/O interface types

• Only supported languages common types
• Common primitives types
• Bool, integer(8,16,32,64bit), float, double,
  String
• Array of primitives types
• Complex types
• Objects

30
Runtime, Pipeline

• Pipeline description language to setup a running
  application initial connections.

31
Pipeline configuration

• Components
• Remote or local
• Connections
• Simple direct link
• Multicast (Publisher/Subscriber)
• (Temporal) Fusion
• Execution
• Pin initialization values
• Threaded execution

32
Current Features

• Supported Platform
• Linux and Windows
• Supported Languages (with minor restrictions)
• Java
• C/C
• Matlab
• Description languages
• CIDL, interfaces description
• PDCL, static pipeline description
• Temporal Fusion components
• Simple components distribution
• Specification on how to write a language plug-in
• Specification on how to write custom connectors

33
Next Features

• Installation Packages
• Windows installer, debian package
• Pipeline
• Kernel API component for controlling application
  dynamic connections
• Description languages tools
• Semi-Automatic CIDL generator
• Graphical tool for pipeline composition
• Supported Platform
• Other OS
• Supported Languages
• .NET framework
• Connectors
• More connectors plugins

34
OI platform Repository of modalities
35
Summary

• Device Components
• Filter/Translator Components
• Debugger Components (Development)
• Connector Components
• Future Work

36
Device components
37
Device Components
38
Filter/Debugger/Connector Components
39
Filter/TranslatorComponents
Example Int 1 string go
Example Int 1 string zoom 10 string in
40
DebuggerComponents
41
Connector Components
42
Components in Progress (UGLA)

• Abstract Devices
• Gesture Language Technique
• Spatial interaction device
• Pointing / panning / zooming
• Lots of variants
• Earcon device
• Tacton device
• Wrappers
• Phidget OI wrapper
• ARToolkit OI wrapper

43
Futur work (UGLA)

• Generic GPS device
• Smell device
• Physiological sensors
• Context-sensing device
• Audio recognisers
• ambient noise level detector
• Vibration recogniser
• Light level recogniser
• Proximity sensor
• ...

44
OI platform Graphical Editor
45
Content

• Introduction
• Demo
• Roadmap
• Conclusion

46
Introduction

• What? Graphically create an assembly of (OI)
  components
• At first Technical (PDCL) Description
• Objective OI Interface
• Not only a graphical editor but an IDE
• Actually static assembly of components
• Next version dynamic assembly of components
• Why?
• Fast prototyping
• Runtime Testing and Monitoring

47
Demo

• Fast prototyping
• Create assembly
• Run the assembly
• Change Multimodal Interaction

48
Roadmap

• Introspection
• Runtime connection
• Monitoring of the components execution
• Usage data capture
• Link with Conceptual Model
• Multimodal interaction
• Multiple OS support

49
Thank you!