ENVIRONMENTAL AWARENESS

1
ENVIRONMENTAL AWARENESS

• Mahaneeya Raman

2
Automated Activity and Object Detection from
Soldier-worn sensors

• To monitor and assist soldiers
• Monitor environment and soldier entity
• Data acquisition through
• Omni-vision Camera
• Body sensors
• GPS/GIS

3
Agenda

• Brief overview of Omni-vision Camera System
• Brief overview of body sensors
• Overview of GPS/GIS
• Image Stitching procedure and problems
  encountered
• BodyMedia SenseWear sensor system

4
Omni-vision Camera System

• Wide angle lenses (f 1.7 mm 110 degrees)
• Higher image resolution than mirror cameras and
  cheaper ones.
• Cameras daisy chained.
• Just 1 Fire wire cable (signal power)

5
Body sensors to predict physical activity and
emotion

• Physiological data related to human activity and
  emotion
• Physical activity accelerometer signals,
  temperature, heat flux
• Human emotion Skin conductance (Galvanic Skin
  Response), heart rate

6
GPS/GIS

• Map data to physical coordinates
• Simplify the classification problem index into
  previously collected databases
• For visualization purposes need Geographic
  Information System (GIS) data
• Setback extensive GIS data is not free!!

7
Image Stitching steps involved

• Camera Calibration
• Correction for radial and tangential distortion
• Projection onto cylindrical coordinates
• Image stitching
• Take-away from this work
• Problems faced - Parallax error
• Suggestion on how to solve

8
Camera Calibration

• To correct radial distortion and tangential
  distortion in images
• Radial Straight lines in real world appear
  curved in the image plane
• Tangential Image not located on a strict plane
  surface

9
Undistorting image and projection onto
cylindrical coordinates

• Polynomial distortion model
• For aligning the images
• To fit together as a panorama
• Conversion from Cartesian to Cylindrical
  coordinates
• r2 x2 y2
• tan ? y / x

10
Images

• Distorted Image Corrected image and
• projected onto
• cylindrical coordinates

11
Image Stitching

• Load the two undistorted images
• Select points using
• cpselect function in Matlab. Frame1_points and
  Frame2_points saved.
• Determine transformation
• T, between the 2 images,
• x Tx
• Map one image onto the other based on the
  transformation.

12
Image Stitching output

• Final output

13
Image stitching of raw images Parallax error
14
Understanding Parallax error

• Distance related measurement error
• High for images that are close to the camera lens
• Occurs because Same points are located at
  different distances from two camera lenses, in
  both images
• Distances between 2 given points is not the same
  in both images
• Solution
• non-linear mapping has to be done
• Ex. Radial Basis Function method / full planar
  perspective models

15
BodyMedia SenseWear PRO2 Armband

• Body sensor worn on upper right arm
• Timestamp, Memory, Battery indicators
• Connects to PC through USB cable
• Collects through 6 continuous streams of data
  channels
• Stores 30 channels of data

16
BodyMedia SenseWear PRO2 Armband

• 2 axis accelerometer
• Heat flux sensor
• Galvanic Skin Response sensor
• Skin Temperature sensor
• Near-Body Temperature Sensor

17
Sensor signal graph for typing-writing
Skin Temperature
Longitudinal Accelerometer
Heat Flux
Transverse Accelerometer
Galvanic Skin Response
18
Applications of SenseWear
COMPANY APPLICATION
NASA The National Bio-Computation Center measuring the physiological effects of the body during simulated space training and research
Health Management Consultants of Virginia To monitor childrens activity levels
UPMC Energy expenditure and sleep detection to measure physiologic and lifestyle changes of patients with health disorders.
Sandia National Laboratories Support tele-medicine initiatives to cope with the aging population
19
Data Collection

• Pattern of data collected
• Typing writing
• Walking simply sitting
• Playing quake watching a comedy clip
• Sample rate - 4 samples/second
• Video surveillance using fire-i cameras while
  data is being collected
• Timestamp between activities

20
Recorded video using 2 fire-i cameras
21
Sensor signal graph for typing-writing, walking -
sitting, playing quake-relaxing
Longitudinal Accelerometer
Transverse Accelerometer
Heat Flux
22
Data Analysis

• Generate signal graph and excel sheets using
  bodymedias innerview software
• Notice Accelerometer signals change considerably
• Use a classifier algorithm
• 50 data training
• 50 data testing

23
KNN and LDA classifier considered Algorithms
for comparison

• KNN K Nearest Neighbor Algorithm
• If x to be classified, and (x1, y1), . . . ,
  (xk, yk) are xs k nearest neighbors, and d(x,
  xi) distance between x and xi, x is classified
  into the nearest neighbor cluster.
• LDA
• Method to find linear discriminant boundaries
  between K classes
• Define K linear discriminant functions for K
  classes
• Classify x to the class with the largest value
  for its discriminant function

24
Conclusion

• Useful GIS data can be integrated with GPS data
  for effective localization and environment
  analysis
• Parallax error can be solved by applying a
  non-linear transformation like RBF or full planar
  perspective models
• Pending - Compare classifier algorithms to
  classify activity, which can be extended to
  predict human emotion as well

25
References

• 1 CAMEO The Camera Assisted Meeting Event
  Observer Paul E. Rybski, Fernando de la Torre,
  Raju Patil, Carlos Vallespi, Manuela Veloso,
  Brett Browning.
• 2 Image Warping - Mikkel B. Stegmann ,
  Informatics and Mathematical Modelling, Technical
  University of Denmark.
• 3 Creating Full View Panoramic Image Mosaics
  and Environment Maps - Richard Szeliski and
  Heung-Yeung Shum, Microsoft Research.
• 4 16 papers on bodymedia applications -
  http//www.bodymedia.com/research/whitepapers.jsp

26
Thank you
27
Matlab toolbox for calibration

• Intrinsic calibration focal length for each
  image axis, an image center, 3 terms of radial
  distortion, and 2 terms of tangential distortion.
• Checker-board method straight lines with easily
  localizable end points and interior points can be
  found in several orientations throughout the
  image plane.

28
Matlab toolbox for calibration

• Images of checkerboard at different inclinations
  (horizontal, vertical, diagonal)
• Provide size of the square
• Extract grid corners of all images, one by one.
• Provide size of window of squares chosen
• Corner extraction - verification

29
Distortion Model Plumb Bob model

• P point in space coordinate vector,
• XXc Xc Yc Zc in camera ref. frame
• Project P on the image plane according to
  intrinsic parameters (fc, cc, alpha_c, kc)
• xn normalized (pinhole) image projection,
• xn Xc/ Zc Yc/ Zc x y
• Let r2 x2 y2

30
Distortion Model Plumb Bob model

• After lens distortion, the new normalized point
  coordinate, xd xd(1) xd(2)
• xd (1 kc(1)r2 kc(2)r4 kc(5)r6 )xn dx
• Where, dx tangential distortion vector
• dx 2kc(3)xy kc(4)(r2 2x2)
• kc(3)(r2 2y2) 2kc(4)xy

31
Distortion Model

• The final pixel coordinates, x_pixel xpyp
• Xp fc(1)(Xd (1) alpha_cXd (2)) cc(1)
• Yp fc(2) Xd (2) cc(2)
• Therefore, xpyp1 KK xd(1)xd(2)1
• where, KK fc(1) alpha_cfc(1) cc(1)
• 0 fc(1) cc(1)
• 0 0 1

32
Image Stitching

• Load 2 images, distortion corrected and projected
  onto cylindrical coordinates.
• Select points using cpselect function in Matlab.
  Input_points and base_points saved.
• Determine transformation T, between the 2 images,
• x Tx
• x s cosa s sina tx x
• y -s sina s cosa ty y
• 1 0 0 1 1

33
Image Stitching - continued

• By rearranging the equation so the warping
  parameters is the vector t in,
• x Zt
• x x y 1 0 0 s cosa
• y y -x 0 1 0 s sina
• 1 0 0 0 0 1 tx
• ty
• 1

34
Radial Basis Function method

• The input layer is the set of source nodes
  (sensory units).
• The second layer is a hidden layer of high
  dimension.
• The output layer gives the response of the
  network to the activation patterns applied to the
  input layer.
• The transformation from the input space to the
  hidden-unit space is nonlinear.
• On the other hand, the transformation from the
  hidden space to the output space is linear.

35
Benefits of SenseWear

• Detect energy expenditure during certain
  non-ambulatory activities
• Detect increased effort and energy expenditure
  associated with load carrying.
• Measure heat produced by the body as a result of
  basic metabolism and, as well as, all forms of
  physical activity.
• Small, unobtrusive, and comfortable to wear.
• It is not invasive and does not alter normal
  patterns of motion or activity