Multimodal sensors - PowerPoint PPT Presentation


PPT – Multimodal sensors PowerPoint presentation | free to download - id: 403f47-MGRhM


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation

Multimodal sensors


Multimodal sensors & digital interfaces Credits The original Multimodal Project developed by and credit for: Zhigang Zhu and Weihong Li (Integration of Laser ... – PowerPoint PPT presentation

Number of Views:29
Avg rating:3.0/5.0
Slides: 23
Provided by: Tao75


Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: Multimodal sensors

Multimodal sensors digital interfaces
  • The original Multimodal Project developed by and
    credit for
  • Zhigang Zhu and Weihong Li (Integration of Laser
    Vibrometry with Infrared Video for Multimedia
    Surveillance Display)

  • Multimodal System Overview
  • Multimedia Sensors
  • Infrared camera
  • The LDV sensor
  • PTZ camera
  • Multimodal System Components
  • System Design Concept
  • Design Issues
  • Integration Issues

Multimodal System Overview
  • The object of this system is to provide a
    multimodal integration of audio, visible, thermal
    for human signature detection.
  • The goal is to use the sensing technologies for
    perimeter surveillance.
  • Sensors, alarm, response.
  • Multimodal system interface
  • The environment, the sensors, and the events.

Multimedia Sensors Infrared Camera
  • Infrared camera
  • FLIR ThermoVision A40M IR camera
  • Temp Range of -20º to 500ºC, accuracy ( of
    reading) 2ºC or 2
  • 320x240 Focal Plane Array
  • 24º FOV Lens
  • Firewire Output IEEE 1394
  • Video output RS170 EIA/NTSC or CCIR/PAL
    composite video for monitoring on a TV screen
  • ThermoVision System Developers Kit (C)
  • Each thermal image is built from 76,800
    individual picture elements that are sampled 60
    times per second by the camera's on-board

Multimedia Sensors Infrared Camera
  • Samples

Figure 1. A person sitting in dark room can be
clearly seen in the IR image. The temperature at
Sp1 on the face is 33.1ºC
Figure 2. Two IR images before and after a person
standing at about 200 feet. The reading of the
temperature at Sp1 changes from 11C to 27C.
Multimedia Sensors Infrared Camera
  • Thermographic measurement techniques
  • An infrared camera measures and images the
    emitted infrared radiation from an object.
  • The radiation measured by the camera does not
    only depend on the temperature of the objects but
    is also a function of the emissivity.
  • Radiation also originates from the surroundings
    and is reflected in the object
  • Radiation from the object and reflected radiation
    will also be influenced by the absorption of the
  • Parameters need to take care
  • The emissivity of the object
  • The reflected temperature
  • The distance between the object and the camera
  • The relative humidity

Multimedia Sensors Infrared Camera
  • Emissivity
  • How much radiation is emitted from the object
  • Object materials and surface treatments exhibit
    emissivity ranging from approximately 0.1 0.95
  • Highly polished (mirror) surface lt 0.1
  • Human skin exhibits an emissivity close to 1
  • Metal low, increase with temperature
  • Non-metal high, decrease with temperature

Multimedia Sensors Infrared Camera
  • Reflected ambient temperature
  • To compensate for the radiation reflected in the
    object and the radiation emitted from the
    atmosphere between the camera and the object.
  • If the emissivity is low, the distance very long
    and the object temperature relatively close to
    that of the ambient it will be important to set
    and compensated for the ambient.
  • Distance
  • The distance between object and the front lens of
    the camera.
  • Relative Humidity
  • Normally, default 50

Multimedia Sensors Infrared Camera
  • History of Infrared Technology
  • Sir William Herschel (1738-1822)
  • Discover of infrared spectrum
  • Marsilio Landriani (1746-1815)
  • As the blackened thermometer was moved slowly
    along the colors of the spectrum, the temperature
    readings showed a steady increase from the violet
    end to the red end.
  • Macedonio Melloni (1798-1854)
  • Rock salt (NaCl) (to be made into lenses and
    prisms) is remarkably transparent to the
  • Sir John Herschel
  • The first heat-picture in 1840, thermograph
  • Samuel P. Langley (1834-1906)
  • Inventor of the bolometer (1880)

Multimedia Sensors LDV sensor
  • Vibrometer types
  • Single Point Vibrometers
  • Measure the vibration of an object in the
    direction of laser beam
  • Differential Vibrometers (dual beam)
  • Allow vibration measurement between two points
    vibrating relative to each other.
  • Rotational Vibrometers
  • Measure angular vibrations on rotating
  • In-plane Vibrometers
  • measure continuous (DC) velocity and superimposed
    variable (AC) components perpendicular to the
    central axis of two converging laser beams.
  • 3D Vibrometers

Multimedia Sensors LDV sensor
  • Laser Doppler Vibrometer (LDV)
  • Optical instruments for accurately measuring
    velocity and displacement of vibrating structures
    completely without contact.
  • Sensor head OFV-505
  • HeNe laser, ? 633.8 nm.
  • OFV-SLR lens (f30mm) 1.8m 200m, auto focus
  • Controller OFV-5000 with a digital velocity
    decode card VD-6
  • RS232 interface for computer control
  • Telescope VIB-A-P05
  • 1º vertical tilt and 1.5º horizontal tilt

Multimedia Sensors LDV sensor
  • Measurement Principle

S is the light source f is frequency P is the
moving with velocity v and reflects the light O
is the receiver (f ?fD)
Resultant frequency shift
For vibrometers SO ("backscatter") ?1 - ?2,
Multimedia Sensors LDV sensor
  • LDV schema
  • Velocity is directly obtained by demodulation ?
  • Voice frequency f 300Hz 3000Hz
  • LDV can detect vibration at a magnitude as low as
    m ?/2?f 1/(23.14300) 0.5µm

Multimedia Sensors PTZ camera
  • Pan/tilt/zoom (PTZ) camera
  • Human and other target detection at a large
  • Canon PTZ
  • 26X optical zoom lens 12X digital zoom
  • Pan 100º, Tilt 90º/-30º
  • Built-in IR light (effective up to 9 feet)
  • BNC video output
  • RS-232 computer control interface

Multimedia Sensors PTZ camera
  • PTZ Samples

Two images of a person at a distance of about 200
feet, captured by changing the zoom factors of
the PTZ camera.
Multimodal System Components
  • Three components
  • The IR/EO imaging video surveillance component
  • Human motion tracking, human face detection
  • Thermal Camera for daytime and nighttime
  • Visible camcorder (sony), Web cam (logitech)
  • The LDV audio surveillance component
  • Audio signal capture, voice recognition
  • The human-computer interaction component
  • Cognitive understanding of the environment, the
    sensors, and the events

System Design Concept
  • The overall goal the project is to design a human
    computer interface for human-centered multimodal
    (MM) surveillance.

Design Issues
  • Issues need to be considered
  • how to use EO camera tracking human motion
  • how to incorporate IR imaging with existing EO
    captured image
  • how to use IR imaging to help the laser Doppler
    vibrometer to select the appropriate targets
  • how to select optimal viewpoint from audio

Integration Issues
  • Target detection and localization via IR/EO
  • Set up an IR/EO imaging system with an IR camera
    and a PTZ camera for finding vibration targets
    for LDV listening
  • Registration between the IR/EO imaging system and
    the LDV system.
  • Two types of sensors need to be precisely aligned
    so that we can point the laser beam of the LDV to
    the target that the IR/EO imaging system has
  • Future research on automated targeting and

  • Main multimodal system technical report
  • http//
  • Polytec Laser Vibrometer
  • http//
  • FLIR Systems Security ThermoVision Cameras
  • http//
  • Paper
  • Z Zhu, W Li, Integrating LDV Audio and IR Video
    for Remote Multimodal Surveillance
  • Others