HiFi: Distributed Sensing and Information Management

1
HiFi Distributed Sensing and Information
Management
Team HiFi Michael Franklin, Shawn Jeffery,
Sailesh Krishnamurthy, Frederick Reiss, Shariq
Rizvi, Eugene Wu, Owen Cooper, Anil Edakkunni,
and Wei Hong
UC Berkeley, Intel Research Berkeley

• Presented by Shawn Jeffery
• jeffery_at_cs.berkeley.edu
• CENTS Retreat 1/14/05

2
Thanks for staying!

• The most snow the Reno-Lake Tahoe area has seen
  since 1916.
• -- AP
• Ideal with nice weather and outrageous
  conditions
• -- Squaw Valley
• The country's BEST snow conditions
• -- Alpine Meadows
• Stellar Conditions
• -- Heavenly

3
Itinerary

• Introduction High Fan-in Systems
• HiFi Overview
• Initial Prototype
• Ongoing Work and Future Directions
• Conclusions

4
Introduction

• Receptors everywhere!
• Wireless sensor networks, RFID technologies,
  digital home, network monitors, ...

• Somehow need to make sense of this data to
  provide near real-time decision support

5
High Fan-in Systems
The Bowtie

• Challenges in 3 dimensions
• Geography
• Time
• Resources

Large numbers of receptors large data volumes
Hierarchical, successive aggregation
6
Supply-Chain Management (SCM)
Headquarters
Regional Centers
Warehouses, Stores
Dock doors, Shelves
Receptors
7
State of the Art

• Not seen as a data management issue
• Focus on protocol design
• Different data models at each level
• Reinventing query languages at each level
• Piecemeal/stovepipe approach
• Each type of receptor (RFID, sensors, etc)
  handled separately
• Current solutions tend to be hand-coded,
  script-based approaches
• ? No end-to-end, integrated solution for managing
  distributed receptor data

8
Itinerary

• Introduction High Fan-in Systems
• HiFi Overview
• Initial Prototype
• Ongoing Work and Future Directions
• Conclusions

9
HiFi Cascading Stream Processing in a High
Fan-in System

• A data management infrastructure for high fan-in
  environments
• Uniform Declarative Framework
• Every node is a data stream processor that speaks
  SQL-ese
• ? stream-oriented queries at all levels
• Hierarchical, stream-based views as an organizing
  principle

10
Why Declarative? (i.e., a lesson in database
dogma)

• Current solutions have a different data model,
  different API at each level
• Simplifies programming
• Independence data, location, platform
• Many optimization opportunities

11
A Brief Aside Data Stream Processing
Result Tuples
Result Tuples
Queries
Queries
Data
Traditional Database
Data Stream Processor

• Data streams are infinite
• Continuous, long running queries
• Real-time processing

12
A Brief Aside Data Stream Processing
A typical streaming query
Window Clause
SELECT S.city, AVG(temp) FROM SOME_STREAM
S range by 5 seconds slide by 5
seconds WHERE S.state California GROUP BY
S.city
I want to look at 5 seconds worth of data
I want a result tuple every 5 seconds
Window
Data Stream

Result Tuple(s)
Result Tuple(s)
13
Hierarchical Query Processing
SELECT S.area, AVG(S.temp) FROM SENSOR_STREAM S
range by 5 sec slide by 5
sec GROUP BY S.area
I provide national monthly values for the US

• Continuous and Streaming
• Windows
• Sharing
• Hierarchical
• Temporal granularity vs. geographic scope

I provide avg weekly values for California
I provide avg daily values for Berkeley
I provide raw readings for Soda Hall
14
Basic HiFi Architecture

• Hierarchical federation of nodes
• Each node
• Data Stream Query Processor (DSQP)
• HiFi Glue
• Views drive system functionality
• Metadata Repository (MDR)

15
HiFi Design Considerations

• Dealing with Real-World Data
• System Management
• Hierarchical Windowed Views with Sharing
• Topological Fluidity
• Query Planning and Data Placement
• Complex Event Processing
• Archiving and Prioritization
• Privacy and Access Control

16
Itinerary

• Introduction High Fan-in Systems
• HiFi Overview
• Initial Prototype
• Ongoing Work and Future Directions
• Conclusions

17
Envisioning HiFi
Building HiFi
18
A Tale of Two Systems

• TelegraphCQ
• Data stream processor
• Continuous, adaptive query
• processing with aggressive sharing
• TinyDB
• Declarative query processing for
• wireless sensor networks
• In-network aggregation

19
Initial Prototype
PC
TelegraphCQ
Stargates
TinyDB
Sensor Networks RFID Readers
RFID Wrappers
20
Initial Prototype
Demoed _at_ VLDB 04, Intel
Research Berkeley Open House
21
HiFi Design Considerations

• Dealing with Real-World Data
• System Management
• Hierarchical Windowed Views with Sharing
• Topological Fluidity
• Query Planning and Data Placement
• Complex Event Processing
• Archiving and Prioritization
• Privacy and Access Control

• Dealing with Real-World Data
• System Management
• Hierarchical Windowed Views with Sharing
• Topological Fluidity
• Query Planning and Data Placement
• Complex Event Processing
• Archiving and Prioritization
• Privacy and Access Control

22
CSAVA Processing RFID Data in HiFi

• RFID data is gross!
• Lost readings
• Errant readings
• Duplicate readings
• Use queries to make the data usable
• CSAVA
• Clean ? Smooth ? Arbitrate ? Validate ?
  Analyze

23
CSAVA Processing RFID Data in HiFi
Clean
CREATE VIEW cleaned_rfid_stream AS (SELECT
receptor_id, tag_id FROM rfid_stream rs WHERE
read_strength gt strength_T)
24
CSAVA Processing RFID Data in HiFi
Smooth
CREATE VIEW smoothed_rfid_stream AS (SELECT
receptor_id, tag_id FROM cleaned_rfid_stream
range by 5 sec, slide by 5
sec GROUP BY receptor_id, tag_id HAVING
count() gt count_T)
Clean
25
CSAVA Processing RFID Data in HiFi
Arbitrate
CREATE VIEW arbitrated_rfid_stream AS (SELECT
receptor_id, tag_id FROM smoothed_rfid_stream rs
range by 5 sec, slide by 5
sec GROUP BY receptor_id, tag_id HAVING
count() gt ALL (SELECT count() FROM
smoothed_rfid_stream range by 5
sec, slide by 5 sec
WHERE tag_id rs.tag_id GROUP BY
receptor_id))
Smooth
Clean
26
CSAVA Processing RFID Data in HiFi
Validate
CREATE VIEW validated_tags AS (SELECT tag_name,
FROM arbitrated_rfid_stream rs range by
5 sec, slide by 5 sec,
known_tag_list tl WHERE tl.tag_id rs.tag_id
Arbitrate
Smooth
Clean
27
CSAVA Processing RFID Data in HiFi
Analyze
CREATE VIEW tag_count AS (SELECT tag_name,
count() FROM validated_tags vt range by
5 min, slide by 1 min GROUP BY
tag_name
Validate
Arbitrate
Smooth
Clean
28
CSAVA Processing RFID Data in HiFi
Analyze
Augment
Augment
Validate
Convert
Convert
Arbitrate
Aggregate
Aggregate
Smooth
Clean
29
CSAVA

• An example of HiFi processing, but instrumental
  in dealing with real world data

Arbitrate
Multiple Receptors
Smooth
Window
Clean
Single Tuple
CSAVA
Generalization
30
System Management

• Our small deployment
• 20 individual devices (4 types of devices)
• 5 different platforms (OS Hardware)
• ? Management nightmare

• System-wide management is crucial
• Both coarse and fine-grained
• Where were headed
• System monitoring needed turn the lens inwards
  to introspect on system state
• Use uniform declarative framework to provide
  failover and load balancing

31
Itinerary

• Introduction High Fan-in Systems
• HiFi Overview
• Initial Prototype
• Ongoing Work and Future Directions
• Conclusions

32
Ongoing Work and Future Directions

• Bridging the physical-digital divide
• Define scope of Virtual Device functionality
• Hierarchical query processing
• Query planning, dissemination
• Complex event processing
• Unify event and data processing
• System deployment and management
• Archiving and prioritization

33
Conclusions

• Receptors everywhere ? High Fan-In Systems
• Uniform declarative framework is the key to
  building these systems
• The HiFi project is exploring this approach
• Our initial prototype
• Leveraged TelegraphCQ and TinyDB
• Validated the HiFi approach
• Identified research directions
• Broad in scope much work to be done!

34
Questions?

• jeffery_at_cs.berkeley.edu
• hifi.cs.berkeley.edu