Speech and Gesture Corpus From Designing to Piloting

1
Speech and Gesture CorpusFrom Designing to
Piloting

• Gheida Shahrour
• Supervised by
• Prof. Martin Russell
• Dr Neil Cooke

• Electronic, Electrical and Computer Engineering
• University of Birmingham

2
Motivation

• Our research focuses on modelling human behaviour
  from body motion.
• No dataset which could serve our research focus.

3
Dataset Specification

• We need data that
• Contains the motion of peoples head, arms and
  hands
• Captured from people come from different cultural
  backgrounds
• Contains spontaneous speech
• Captured using a marker-based tracking technique

4
Why Marker-based Tracking Technique?

• Capturing peoples gestures is mainly based on
  computer vision techniques
• skin colour- peoples skin light in images.
• contour of people- objects may overlap/occluded
• tracking from sequence of frames-may not be
  accurate
• images are from 2D- accuracy issues.
• To Avoid these problems
• We will capture gestures using marker-based
  optical motion tracking
• data obtained from 3D coordinate system
• less occlusion recovered easily
• tracking the object accurately- good calibration
• tracking the light-reflective markers- accuracy.

5
Qualisys Track Manager (QTM)

• The Balance and Posture Laboratory in the School
  of Psychology equipped with QTM system
  (http//www.qualisys.com)
• 12 cameras with LED strobes which emits a beam of
  infrared light which is not visible to the naked
  eye.
• QTM Software Analogue Interface for recording
  speech
• passive markers- different sizes
• calibration Kit axis L shape wand T shape.

http//www.qualisys.com
6
Camera Strobe
http//www.qualisys.com
7
How it works?
.

• The spherical markers are coated with a material
  to amplify their brightness.
• The strobes project light towards the markers and
  the markers reflect it back to the camera
• Then the camera system measures a 2-dimensional
  position of the reflective target by combining
  the 2-D data from several cameras.
• The camera uses the reflected data from multiple
  cameras to calculate the 3D position of the
  markers with high spatial resolution.

8
How it works?
9
The Process of Capturing data
.

• Attach markers on the objects of interest- how?
• Define the measurement area where subjects will
  stand
• Test the area
• Calibrate the area
• Capture your data
• Save your data

10
Reprocess Data Files

• Reprocess the files you captured to construct the
  3D view-how?

11
Labeling Data

• Label your data how?
• Create a text file- Unique name
• Unique colour
• Upload the file
• Drag drop
• Play the motion data
• Play it again
• Fill the gap
• Play it again
• Save the file
• Export the data

12
Experiments (1)_Methods Materials

• 2 volunteers each wears 36 7mm flat-based half
  spherical markers on
• - head(4)
• - elbows(2)
• - waist(4)
• - golf gloves(26).
• 12 cameras measurement volume is not specified
• frame rate 200 frames per second
• speech is not recorded.

13

• Experiments (1)3D View

14

• Experiments (1)Best Result

15
Experiment (2)_Motivation

• To improve the quality of data.
• 1. Quantity number of unidentified markers
  trajectories should be the same number of the
  markers used in the experiment.
• 2. Quality No loss of markers, ghost markers
• The technique the reduction both the number of
  markers the measurement volume

16
Typical 3D Data Cameras Position
17
Low Vs High 3D Tracker Parameters

• Prediction error
• Residual the remaining of the trajectory set to
  low
• Filling gaps between frames

http//www.qualisys.com
18
Markers Trajectories Filling the Gap
19
Missing Data
20
How to Fill these Gaps?
21
Experiments (2)_Methods Materials

• 3 volunteers each wears 28 7mm flat-based half
  spherical markers on
• - head(4)
• - elbows(2)
• - shoulders(2)
• - waist(4)
• - golf gloves(16)

22
Experiments (2)_ Measurement Volume
23
Experiments (1)_Cameras Position
24
Experiments (2)_Cameras Position
25
Experiments (2)_Sessions
26
Experiments (2)_Result
27
Experiments (2)_Result
28
Conclusion

• We will track motion of head, arms and hand
• Leave 3 fingers out middle, ring and pink.
• Occlusion of the markers on fingers is not only
  due to the cameras set up, but also due to the
  degree of freedom of the hands
• Finding unidentified trajectories of markers is
  laborious and time consuming.
• Tracking all fingers is very useful for many
  applications such as Sign Language but this is
  not our focus.

29
Data collection_ assignment

• Each volunteer will wear not less than 12mm
  passive markers on head(4), elbows(2), waist(4),
  shoulder(3) and gloves(10)

30
Group Setup

• Put yourselves into groups of 3.
• The members of each group should be from the same
  first language, same gender same country of
  birth
• Each member in British group (country of birth is
  Britain first language is English) will record
  2 sessions. Each session will last 15 minutes
  captured in 5 stages. Each stage lasts for 3
  minutes.
• Each member in the cultural group (country of
  birth is not Britain first language is not
  English) will record 4 sessions. 2 sessions in
  English as a Second Language and 2 in their first
  language. Each session will last 15 minutes
  captured in 5 stages. Each stage lasts for 3
  minutes.

31

• Any Question?