Stream Hierarchy Data Mining for Sensor Data - PowerPoint PPT Presentation

Loading...

PPT – Stream Hierarchy Data Mining for Sensor Data PowerPoint presentation | free to view - id: 78f301-MTllZ



Loading


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation
Title:

Stream Hierarchy Data Mining for Sensor Data

Description:

From Sensors to Streams An Outline. Data Stream Overview. Data Stream Visualization . Temporal Heat Map. Data Stream Modeling. Extensible Markov Model – PowerPoint PPT presentation

Number of Views:170
Avg rating:3.0/5.0
Slides: 26
Provided by: smu136
Category:

less

Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: Stream Hierarchy Data Mining for Sensor Data


1
Stream Hierarchy Data Mining for Sensor Data
  • Margaret H. Dunham
  • SMU
  • Dallas, Texas 75275
  • mhd_at_engr.smu.edu
  • Vijay Kumar
  • UMKC
  • Kansas City, Missouri 64110
  • kumarv_at_umkc.edu

2
From Sensors to Streams An Outline
  • Data Stream Overview
  • Data Stream Visualization
  • Temporal Heat Map
  • Data Stream Modeling
  • Extensible Markov Model
  • Data Stream Hierarchy

3
From Sensors to Streams An Outline
  • Data Stream Overview
  • Data Stream Visualization
  • Temporal Heat Map
  • Data Stream Modeling
  • Extensible Markov Model
  • Data Stream Hierarchy

4
From Sensors to Streams
  • Data captured and sent by a set of sensors is
    usually referred to as stream data.
  • Real-time sequence of encoded signals which
    contain desired information. It is continuous,
    ordered (implicitly by arrival time or explicitly
    by timestamp or by geographic coordinates)
    sequence of items
  • Stream data is infinite - the data keeps coming.

5
Data Stream Management Systems (DSMS)
  • Software to facilitate querying and managing
    stream data.
  • Retrieve the most recent information from the
    stream
  • Data aggregation facilitates merging together
    multiple streams
  • Modeling stream data to summarize stream
  • Visualization needed to observe in real-time the
    spatial and temporal patterns and trends hidden
    in the data.

6
DSMS Problems
  • Stream Management development in state similar to
    that of databases prior to 1970s
  • Each system/researcher looks at specific
    application or system
  • No standards concerning functionality
  • No standard query language
  • Unreasonable to expect end users will access raw
    data, data in the DSMS, or even data at a
    summarized view
  • Domain experts need to see a higher level of
    data

7
Our Proposal
  • Four level data abstraction to facilitate the
    creation of actionable intelligence for domain
    experts evaluating sensor data.

8
From Sensors to Streams An Outline
  • Data Stream Overview
  • Data Stream Visualization
  • Temporal Heat Map
  • Data Stream Modeling
  • Extensible Markov Model
  • Data Stream Hierarchy

9
Assumptions for Our Research
  • End User
  • May not be knowledgeable concerning sensors
  • Probably a Domain Expert
  • May not need to see exact sensor values
  • Concerned with trends and approximate values
  • Need to see data from MANY sensors at one time
  • Need to see data continuously in a visualization
    of the stream

10
Suppose There Were MANY Sensors
  • Traditional line graphs would be very difficult
    to read
  • Requirements for new visualization technique
  • High level summary of data
  • Handle multiple sensors at once
  • Continuous
  • Temporal
  • Spatial

11
Temporal Heat Map
  • Also called Temporal Chaos Game Representation
    (TCGR)
  • Temporal Heat Map (THM) is a visualization
    technique for streaming data derived from
    multiple sensors.
  • It is a two dimensional structure similar to an
    infinite table.
  • Each row of the table is associated with one
    sensor value.
  • Each column of the table is associated with a
    point in time.
  • Each cell within the THM is a color
    representation of the sensor value
  • Colors normalized (in our examples)
  • 0 While
  • 0.5 Blue
  • 1.0 - Red

12
Cisco Internal VoIP Traffic Data
  • Values ?
  • Complete Stream CiscoEMM.png
  • VoIP traffic data was provided by Cisco Systems
    and represents logged VoIP traffic in their
    Richardson, Texas facility from Mon Sep 22
    121732 2003 to Mon Nov 17 112911 2003.
  • Time ?

13
Derwent River (UK)
Derwent Temporal Heat Map derwentrotate.png
14
From Sensors to Streams An Outline
  • Data Stream Overview
  • Data Stream Visualization
  • Temporal Heat Map
  • Data Stream Modeling
  • Extensible Markov Model
  • Data Stream Hierarchy

15
Data Stream Modeling Requirements
  • Summarization (Synopsis )of data
  • Use data NOT SAMPLE
  • Temporal and Spatial
  • Dynamic
  • Continuous (infinite stream)
  • Learn
  • Forget
  • Sublinear growth rate - Clustering

16
Extensible Markov Model
  • Extensible Markov Model (EMM) at any time t, EMM
    consists of a Markov Chain with designated
    current node, Nn, and algorithms to modify it,
    where algorithms include
  • EMMCluster, which defines a technique for
    matching between input data at time t 1 and
    existing states in the MC at time t.
  • EMMIncrement algorithm, which updates MC at time
    t 1 given the MC at time t and clustering
    measure result at time t 1.
  • EMMDecrement algorithm, which removes nodes from
    the EMM when needed.
  •  In addition, the EMM has associated Data Mining
    functions such a Rare Event Detection and
    Prediction
  • Jie Huang, Yu Meng, and Margaret H. Dunham,
    Extensible Markov Model, Proceedings IEEE ICDM
    Conference, November 2004, pp 371-374.

17
EMM Learning
  • lt18,10,3,3,1,0,0gt
  • lt17,10,2,3,1,0,0gt
  • lt16,9,2,3,1,0,0gt
  • lt14,8,2,3,1,0,0gt
  • lt14,8,2,3,0,0,0gt
  • lt18,10,3,3,1,1,0.gt

18
EMM Forgetting
19
EMM Sublinear Growth Rate
Minnesota Department of Transportation (MnDot)
20
From Sensors to Streams An Outline
  • Data Stream Overview
  • Data Stream Visualization
  • Temporal Heat Map
  • Data Stream Modeling
  • Extensible Markov Model
  • Data Stream Hierarchy

21
Traditional DBMS Data Abstraction
  • Three levels of data abstraction
  • Physical,
  • Logical
  • External
  • Data is normally pulled to the user by a query

22
Proposed DSMS Data Abstraction
  • Abstraction
  • Level 0 - Physical Level
  • Raw data from sensors
  • Cannot be stored
  • Level 1 DSMS
  • Sensor data is merged, aggregated, and cleansed.
  • DSMS queries may be processed against this data.
  • Level 2 Model
  • Summarization (Synopsis )of data
  • Level 3 Domain Expert
  • Summary Visualization
  • Data is normally pushed to the user

23
Levels Lowest Level Highest Level Abstraction Inter-level Data Migration
Memory Hierarchy n External Storage Subset/Cache/Buffer Fetch/Prefetch
DBMS Data Hierarchy 3 Physical Storage External View Fetch, Prefetch
Data Warehouse n Operational Data Cube/ Multidimensional View Aggregation
Stream Hierarchy 4 Sensor Data Visualization/Triggers Automatic Push
24
(No Transcript)
25
Stream Hierarchy Summary
  • Except for the inter-level functionality
    requirements, each level functionality is
    independent of the others and may differ across
    different implementations.
  • The model used must capture time and ordering of
    data, be able to both learn and forget, and use
    some variation of clustering.
  • Visualization at the domain expert level must
    capture both time and ordering. It addition it
    should be able to be easily read for many sets
    of sensors.
About PowerShow.com