Desiging a Virtual Information Telescope using Mobile Phones and Social Participation Romit Roy Chou

1
Desiging a Virtual Information Telescopeusing
Mobile Phones and Social ParticipationRomi
t Roy ChoudhuryAsst. Prof. (Duke University)
2

• A little bit about ourselves

3
Webpage
http//synrg.ee.duke.edu
SyNRG
4
Our Research
What are the visions (top down)
Ubiquitous Services
Incentives
Application
Privacy
Security
Eavesdropping
Loss Discrimination
Transport
Mobility
Network
Energy Savings
MAC / Link
Spatial Reuse
Interference Mgmt.
PHY
Channel fluctuations
What can be enabled (bottom up)
5
Some Ongoing Projects
Information Telescope
Spotlight
MobiSys 2008 Infocom 09
ICNP 2008
Hotnets 2008
Shuffle
SmartCast
6
Todays Talk
Information Telescope
Papers Infocom 2009 HotMobile 2009 MobiSys
2008 Posters, Demos MobiCom, Sensys,
MobiSys Mobicom Student Research Contest Award
2008
7
Todays Talk
Information Telescope
System and Applications
Ongoing, Future Work
Challenges/Opporunities
Vision

• EnLoc
• SurroundSense
• CacheCloak

8

• Virtual Information Telescope

9
Context

• Next generation mobile phones will have
• large number of sensors
• Cameras, microphones, accelerometers, GPS,
• compasses, health monitors,

10
Context

• Each phone may be viewed as
• a micro lens
• Exposing a micro view of the physical world
• to the Internet

11
Context

• With 3 billion active phones
• in the world today
• (the fastest growing comuting platform )
• Our Vision is

12
A Virtual Information Telescope
Internet
13

• One instantiation of this vision through
• a system called Micro-Blog

- Content sharing - Content querying - Content
floating
14
Content Sharing
Virtual Telescope
Web Service
Cellular, WiFi
Visualization Service
People
Phones
Physical Space
15
Content Querying
Virtual Telescope
Web Service
Cellular, WiFi
Visualization Service
People
Phones
Physical Space
16
Content Floating on physical space
superb sushi
Safe_at_ Nite?
17

• If designed carefully, a variety of
• applications may emerge on Micro-Blog

18
Applications

• Tourism
• View multimedia blogs query for specifics
• Micro Reporters
• News service with feeds from individuals
• On-the-fly Ride Sharing
• Ride givers advertize intension w/ space-time
  sticky notes
• Respond to sticky notes once you arrive there
• Virtual order on physical disorder
• Land in a new place, and get step by step
  information
• RSS Feeds on Location
• Inform me when a live band is playing at the mall

19
MiroBlog Prototype

• Nokia N95 phones
• Symbian platform
• Carbide C code

20

• Micro-Blog Beta live at
• http//synrg.ee.duke.edu/microblog.html

21
Prototype
22
Case Studies

• Micro-Blog phones distributed to volunteers
• 12 volunteers
• 4 phones in 3 rounds
• 3 weeks
• Not great UI
• Basic training for users
• Exit interview revealed
• useful observations

23
From Exit Interview

• Fun activity for free time
• Needs much cooler GUI
• Privacy control vital, dont care about
  incentives
• more interesting to reply to questions
  interested in knowing who is asking
• Voice is personal, text is impersonal
• Easier to correct text audio blogs easier but
  
• Logs show most blogs between 500 to 900pm
• Probably better for battery usage as well

24
Thoughts

• Micro-Blog
• Rich space for applications and services

25

• Several research challenges and opportunities
• Energy-efficient localization
• Symbolic localization through ambience sensing
• Location privacy
• Incentives
• Spam
• Information distillation
• User Inerfacing

Our Research
26
Disclaimer

• All of our projects are ongoing,
• hence not fully mature
• Todays talk more about the problems
• than about solutions

27

• Problem I
• Energy Efficient Localization
• (EnLoc)

28
To GPS or not to GPS

• GPS is popular localization scheme
• Good error characteristics 10m
• Apps naturally assume GPS
• Shockingly, first Micro-Blog demo lasted lt 10
  hours

29
Cost of Localization

• Performed extensive measurements
• GPS consumes 400 mW, AGPS marginally better
• Idle power consumption 55 mW

30
Alternate Localization

• WiFi fingerprinting, GSM triangulation
• Place Lab, SkyHook
• Improved energy savings
• WiFi 20 hours
• GSM 40 hours
• At the cost of accuracy
• 40m
• 200m

31
Tradeoff Summary
40
20
200
32
Formulation
L(t2) L(t3) L(t4)
L(t0)
L(t1)
L(t6)
L(t5)
L(t7)
Error
t0
t1
t2
t3
t4
t5
t6
t7
Given energy budget, E, Trace T, and location
reading costs, egps , ewifi , egsm Schedule
location readings to minimize avg. error
33
Dynamic Program

• Minimize the area under the curve
• By cutting the curve at appropriate points
• Number of (GPS WiFi GSM) cuts must cost lt
  budget

34

• Offline optimal offers lower bound on error
• Online algorithm necessary
• Online optimal difficult
• Need to design heuristics

35
Our Approach

• Do not invest energy if you can
• predict (even partially)

36
Predictive Heuristics

• Prediction opportunities exist
• Human users are not in brownian motion (exploit
  inertia)
• Exploit habitual mobility patterns
• Population distribution can be leveraged
• Prediction also incorporated into Dynamic Program
• Optimal computed on a given predictor

Error
Prediction generates different error curve
t0
t1
t2
t3
t4
t5
t6
t6
37
Mobility Profiling

• Build logical mobility tree per-user
• Each link an uncertainty point (UP)
• Sample location only when uncertain
• Location predictable between UPs
• Exploit acclerometers
• Predict traffic turns
• Periodically localize to reset errors

38
Population Statistics

• Humans may deviate from mobility profile
• Predict based on population statistics

39
Buy Accuracy with Energy

• Comparison of optimal with simple interpolation
• GPS clearly not the right choice

40
Thoughts

• Localization cannot be taken for granted
• Critical tradeoff between energy and accuracy
• Substantial room for saving energy
• While sustaining reasonably good accuracy
• However, physical localization
• May not be the way to go
• Several motivations to pursue symbolic
  localization

41

• Problem 2
• Symbolic localization
• (SurroundSense)

42
Symbolic Localization

• Services may not care about physical location
• Symbolic location often sufficient
• E.g., coffee shop, movie, park, in-car
• Physical to Symbolic conversion
• Lookup location name based on GPS coordinate
• However, risky

RadioShack
Starbucks
GPS Error range
43
Hypothesis
Its possible to localize phones by sensing the
ambience such as sound, light, color, movement,
orientation
44
SurroundSense

• Develop multi-modal fingerprint
• Using ambient sound/light/color/movement etc.

Starbucks
RadioShack
Wall
SurroundSense Server
45
SurroundSense

• Each individual sensor not discriminating enough
• Together, they are quite unique
• Use Support Vector Machines to identify
  uniqueness

Location
Classification Algorithm (SVM)
Fingerprint Database
46
Should Ambiences be Unique Worldwide?
GSM provides macro location (mall) SurroundSense
refines to Starbucks
47
Why will it work?

• Economics forces nearby businesses to be
  different
• Not profitable to have 5 chinese restaurants
• with same lighting, music, color, layout, etc.
• SurroundSense exploits this ambience diversity

The Intuition
48
Fingerprints

• Sound
• Color

49
Fingerprints

• Light
• Movement

50
Ambience Fingerprinting
Fingerprint Filtering Matching
Sound
Color/Light

Compass

RF/Acc.
Logical Location
Fingerprint Database
Macro Location
Candidate Fingerprints
51
Full System on Nokia N95

• Experimented on 100 stores
• 10 different clusters
• Different parts of Duke campus
• and in Durham city

52
Full System on Nokia N95

• Some classifications were incorrect
• But we wanted to know how much incorrect?
• We plotted Top-K accuracy
• Top-3 accuracy proved to be 100 for all stores

53
Issues and Opportunity

• Cameras may be inside pockets
• Now, we detect when its taken out
• Activate cameras, and take pictures
• Future phones will be flexible (wrist watch) -
  see Nokia Morph
• Electroic compasses can fingerprint layout
• Tables and shelves laid out in different
  orientations
• Users forced to orient in those ways

54
Summary

• Ambience can be a great clue about location
• Ambient Sound, light, color, movement
• None of the individual sensors good enough
• Combined they may be unique
• Uniqueness facilitated by economic incentive
• Businesses benefit if they are mutually diverse
  in ambience
• Ambience diversity helps SurroundSense

55

• Problem 3
• Location Privacy
• (CacheCloak)

56
Location Privacy

• Problem
• Research

Continuous location exposure a serious threat to
privacy
Preserve privacy without sacrificing the quality
of continuous loc. based apps
57
Just Call Yourself Freddy

• Pseudonymns
• Effective only when infrequent location exposure
• Else, spatio-temporal patterns enough to
  deanonymize
• think breadcrumbs

Leslie
Jack
John
Susan
Alex
Romits Office
58
Add Noise

• K-anonymity
• Convert location to a space-time bounding box
• Ensure K users in the box
• Location Apps reply to boxed region
• Issues
• Poor quality of location
• Degrades in sparse regions
• Not real-time

Bounding Box
You
K4
59
Confuse Via Mixing

• Path intersections is an opportunity for privacy
• If users intersect in space-time, cannot say who
  is who later
• Issues
• Users may not be collocated in space and time
• Mixing still possible at the expense of delay

60

• Existing solutions seem to suggest
• Privacy and Quality of Localization (QoL)
• is a zero sum game
• Need to sacrifice one to gain the other

61
Our Goal

• Break away from this tradeoff
• Target Spatial accuracy
• Real-time updates
• Privacy guarantees
• Even in sparse populations

We design CacheCloak
62
CacheCloak Intuition

• Exploit mobility prediction to create
• future path intersections
• Users paths are like crossroads of breadcrumbs
• App knows precise locations, but doesnt know the
  user

63
CacheCloak

• Assume trusted privacy provider
• Reveal location to CacheCloak
• CacheCloak exposes anonymized location to Loc. App

Loc. App1
Loc. App2
Loc. App3
Loc. App4
CacheCloak
64
CacheCloak Design

• User A drives down path P1
• P1 is a sequence of locations
• CacheCloak has cached response for each location
• User A takes a new turn (no cached response)
• CacheCloak predicts mobility
• Deliberately intersects predicted path with
  another path P2
• Exposes predicted path to application
• Application replies to queries for entire path
• CacheCloak always knows users current location
• Forwards cached responses for that precise
  location

65
CacheCloak Design

• Adversary confused
• New path intersects paths P1 and P2 (crossroads)
• Not clear where the user came from or turned onto
• Example

66
Example
67
Benefits

• Real-time
• Response ready when user
• arrives at predicted location
• High QoL
• Responses can be specific to location
• Overhead on the wired backbone (caching helps)
• Entropy guarantees
• Entropy increases at traffic intersections
• In low regions, desired entropy possible via
  false branching
• Sparse population
• Can be handled with dummy users

68
Quantifying Privacy

• City converted into grid of small sqaures
  (pixels)
• Users are located at a pixel at a given time
• Each pixel associated with 8x8 matrix
• Element (x, y) probability that user enters x
  and exits y
• Probabilities diffuse
• At intersections
• Over time
• Privacy entropy

y
x
pixel
69
Diffusion

• Probability of users presence diffuses
• Diffusion gradient computed based on history
• i.e., what fraction of users take right turn at
  this intersection

Time t1
Time t2
Time t3
Road Intersection
70
Evaluation

• Trace based simulation
• VanetMobiSim US Census Bureau trace data
• Durham map with traffic lights, speed limits,
  etc.
• Vehicles follow Google map paths
• Performs collision avoidance

6km x 6km 10m x 10m pixel 1000 cars
71
Results

• High average entropy
• Quite insensitive to user density (good for
  sparse regions)
• Minimum entropy reasonably high

72
Results

• Per-user entropy
• Increases quickly over time
• No user starves of location privacy

73
Issues and Limitations

• CacheCloak overhead
• Application replies to lots of queries
• However, overhead on wired infrastructure
• Caching reduces this overhead significantly
• CacheCloak assumes same, indistinguishable query
• If user asks different query at each road segment
• Overhead increases
• Adaptive branching dummy users
• Offer user-specified privacy guarantee

74
Closing Thoughts

• Two nodes may intersect in space but not in time
• Mixing not possible
• Mobility prediction allows space-time
  intersections
• Enables better privacy

75
Conclusion

• The Virtual Information Telescope
• A generalization of mobile, location
• based, social computing
• Just developing apps
• Not enough
• Many challenges
• Energy
• Localization
• Privacy
• Incentives, data distillation

Internet
76
Conclusion

• Project Micro-Blog
• Addressing the challenges systematically
• Building a fully functional system with
  applications
• The project snapshot as of today, includes

Micro-Blog Overall system and application EnLoc
Energy Efficient Localization SurroundSense
Context aware localization CacheCloak Location
privacy via mobility prediction
77
PhonePoint Pens

• Using phone accelerometers
• To write short messages in the air

78
PhonePoint Pen

• Using phone accelerometers
• To write short messages in the air

Contact Sandip Agrawal sandip.agrawal_at_duke.edu
For details, visit synrg.ee.duke.edu
79

• Please stay tuned for more at
• http//synrg.ee.duke.edu
• Thank You

80
Example
81
Results

• High average entropy
• Quite insensitive to user density (good for
  sparse regions)
• Minimum entropy reasonably high

82
Results

• Length of predictions
• Remains reasonably short
• Overhead proportional to this length

83
Fingerprints

• Acoustic fingerprinting
• Coffee machine hum in cafes, different from A/C
  machine drones in grocery, different from spoon
  clink in restaurants
• Loudness varies (some places with loud music,
  others quiet)

84
Fingerprints

• Color and Light Fingerprints
• Stores have a specific décor (color, light)
• Theme reflected in wall color, floor color,
  lighting style

85
Our Approach

• Build symbolic localization algorithms
• Use low accuracy physical localization as
  baseline
• Low accuracy conserves energy

86
SurroundSense

• Sense ambient light, sound, colors
• Combine sensor readings to generate soft
  fingerprint
• For localization, gather fingerprint from mobile
• Match with database of fingerprints
• Of course, fingerprint may not be globally unique
• Use rough physical localization as a pre-filter

GSM physical location says you are in the
mall SurroundSense augments that with you are
in Apple Store
87

• Sample sound/light shows diversity
• Extract features from signals
• Fingerprint 46 sound 2 light features
• 48 dimensional space

Sound (Frequency domain)
Light intensity
88

• Match test fingerprint with trained database
• Use Nearest Neighbor algorithm for
  classification

Feature Extract
Location
Feature Extract
48 features
89
SurroundSense Design

• Prototype on Tmote Invent sensors
• Sound and light sensors
• Low acoustic frequency range 20, 250 Hz
• Currently porting on Nokia N95 phones

90
Fingerprint Extraction

• Light intensity and sound signals recorded
• Fourier transform on sound
• Overlapping frequency blocks generated
• Each block 23 bands, 10 Hz each
• Extract simple features
• Each 10 Hz band one feature
• Variance of each band another feature
• Normalized light intensity another feature
• Total - 48 features
• Train the system with half the data
• 48 dimensional fingerprint

91
Fingerprint Generation and Matching

• Match test fingerprint with trained database
• Use Nearest Neighbor algorithm for
  classification

Feature Extract
Location
Feature Extract
48 features
92
Results

• SurroundSense offers consistent localization
• Database contains nearby shops in Duke campus
• Both sensors gt sound gt light

Pairwise Similarity
93

• Symbolic localization can be augmented with
• phone accelerometers
• Additional benefits in activity recognition

94
Hypothesis

• Movement partially indicative of location
• People sit in cafes
• Run in gyms
• Walk up/down aisles in grocery stores
• Time duration spent may depend on location
• Augment location accuracy with acc. signatures
• Enable activity recognition as well
• E.g., Advertize shoes to users running in the gym

95
AAMPL

• Acclerometer augmented phone localization
• Train database with many acc. signatures
• Mobile phone 3-axis accelerometer in pockets
• Less useful if phone in ladys handbag
• Use acc. signatures from individuals in real time
• Match against database of signatures
• Localize phone
• Predict activity
• Combine accelerometers with sound and light --
  fingerprint

96
AAMPL Architecture

• Client Server based
• Two-stage classifier
• Phones classify sitting/standing
• Sends to server
• Saves energy
• Server classifies location
• WiFi based basic location
• Augmented by AAMPL
• Server compares classified location with Googles
  result

97
AAMPL Classification
98
Evaluation

• Gathered acc. signatures from many restaurants
• Classified with AAMPL, compared with Google

Store Apple Wholefoods Journeys Solstice
Fast Food Chipotle Jimmy Johns
Restaurant Chais Verde Rockfish
99
Evaluation

• AAMPL classified each location
• Compared corresponding location with Google

100
Thoughts

• Main Idea is that the surrounding is a
  fingerprint.
• Effective for separating out nearby contexts.
• In reality,
• Spatially clustered shops are diverse by design
• SurroundSense exploits this diversity

101
Mote based Prototype

• Used Tmote sensors on behalf of phones
• Equipped with basic
• light sensor and microphone
• Mobile phones expected to be more powerful
• Better audio sensing (Larger freq. range than
  20-250hz)
• Better light sensing with camera

102
Results on TMotes

• Tested in Duke campus (shops restaurants)

Correct matches
SurroundSense identified each store correctly
103
1. Simple Interpolation

• Take GPS reading, followed by WiFi
• Extrapolate GPS in the direction of WiFi
• Reset prediction after threshold time, take GPS
  again