The Vision and Reality of Ubiquitous Computing

1
The Vision and Reality of Ubiquitous Computing

• Prof. Henning Schulzrinne
• Dept. of Computer Science
• Columbia University
• (with Arezu Moghadam, Ron Shacham, Suman
  Srinivasan, Xiaotao Wu and other IRT members
  parts in cooperation with DoCoMo Eurolabs)

2
Overview

• The original vision of ubiquitous computing
• User challenges
• Beyond terminal mobility
• Location as new core service
• Universality 7DS

3
Ubiquitous computing ? ubiquitous communications

• It is invisible, everywhere computing that does
  not live on a personal device of any sort, but is
  in the woodwork everywhere.
• Weisers original vision (Nomadic Issues in
  Ubiquitous Computing, 1996)
• one person, many computers
• many computers embedded in environment
• dynamic ownership
• PC phonebooth
• IR use will grow rapidly
• Updated version, 2007
• not physically invisible, but transparent
• emphasis on communications, not computing
• most devices are mobile (or nomadic)
• cheap electronics ?personal devices
• radio (channelized and UWB)

4
Overview

• The original vision of ubiquitous computing
• User challenges
• Beyond terminal mobility
• Location as new core service
• Universality 7DS

5
User challenges vs. research challenges

• Are we addressing real user needs?
• Engineering vs. sports
• My guesses

ease of use
no manual
reliability
no re-entry no duplication
integration
phishing data loss
cost
limited risk
6
Example Email configuration

• Application configuration for (mobile) devices
  painful
• SMTP port 25 vs. 587
• IMAP vs. POP
• TLS vs. SSL vs. secure authentication
• Worse for SIP...

7
Example SIP configuration
partially explains

• highly technical parameters, with differing names
• inconsistent conventions for user and realm
• made worse by limited end systems (configure by
  multi-tap)
• usually fails with some cryptic error message and
  no indication which parameter
• out-of-box experience not good

8
Mobile whys

• Not research, but examples of real annoyances
• Why does each mobile device need its own power
  supply?
• Why do I have to adjust the clock on my camera
  each time I travel?
• Why do I have to know what my IMAP server is and
  whether it uses TLS or SSL?
• Why do I have to type in my address book?
• Why do I have to synchronize my PDA?
• Why do I have to manually update software?
• Why is connecting a laptop to a projector a
  gamble?
• Why do we use USB memory sticks when all laptops
  have 802.11b?

9
Consumer wireless mobile devices
Prius key
Garage door opener
TAN display
Water leak alarm
wireless door bell
MSN Direct weather
10
Mobile systems - reality
GPS

• idea special purpose (phone) --gt universal
  communicator
• idea is easy...
• mobile equipment laptop phone
• sufficiently different UI and capabilities
• we all know the ideal (converged) cell phone
• difficulty is not technology, but integration and
  programmability
• (almost) each phone has a different flavor of OS
• doesnt implement all functionality in Java APIs
• no dominant vendor (see UNIX/Linux vs. Microsoft)
• external interfaces crippled or unavailable
• e.g., phone book access

location data
11
Example displays and speakers
12
Philosophy transition
PC era cell phone era
One computer/phone, many users
One computer/phone, one user
mainframe era home phone party line
Many computers/phones, one user
ubiquitous computing
anywhere, any time any media
right place (device), right time, right media
13
The mobile ubiquitous challenge
Mobile phone
Mobile Internet access
Interconnected devices
Internet of things
14
What do we need?

• Standards, not new technology
• Radio connectivity
• 802.11a/b/g/n, 802.15.4 ?
• better discovery of networks
• Location information everywhere
• Discovery devices services
• network-local discovery via Bonjour (mDNS) ?
• missing location-based discovery
• Advanced mobility session, personal, service
• Event notification
• Data formats
• location, sensor events, ...

15
Examples of invisible behavior

• MP3 player in car automatically picks up new
  files in home server
• A new email with vcard attachment automatically
  updates my cell phone address book
• The display of my laptop appears on the local
  projector
• without cable or configuration
• I can call people I just met at Mobiquitous
• without exchanging business cards
• My car key opens my front door
• My cell phone serves as a TAN (one-time password)
  generator
• My cell phone automatically turns itself off
  during a lecture
• My camera knows where the picture was taken

16
An interconnected system
opens doors
generates TAN
incoming call
updates location
time, location
address book
alert, events
any weather service school closings
acoustic alerts
17
Thinking beyond 802.11 and UMTS

• Many interesting networks beyond those covered in
  conferences
• ease of access by researchers vs. importance
• 90 of papers on 802.11b and maybe GPRS,
  BlueTooth
• New wireless networks
• broadcast instead of unicast -- useful for many
  ubiquitous applications
• S5 for low-rate sensors (city scale)
• Zigbee (802.15.4) for local sensors (20 - 250
  kb/s)
• FM subcarrier (not really new) -- MSN Direct
• FM Radio Data System -- TMC
• Sirius / XM
• HD radio
• paging

18
Overview

• The original vision of ubiquitous computing
• User challenges
• Beyond terminal mobility
• Location as new core service
• Universality 7DS

19
Application-layer mobility

• terminal mobility
• one terminal, multiple network addresses
• Personal mobility
• one person, multiple terminals
• e.g., Grandcentral
• session mobility
• one user, multiple terminals in sequence or in
  parallel
• service mobility
• services move with user

20
Session mobility

• Walk into office, switch from cell phone to desk
  phone
• call transfer problem ? SIP REFER
• related problem split session across end devices
• e.g., wall display desk phone PC for
  collaborative application
• assume devices (or stand-ins) are SIP-enabled
• third-party call control

R. Shacham, H. Schulzrinne, S. Thakolsri, W.
Kellerer, Ubiquitous device personalization and
use The next generation of IP multimedia
communications, ACM TOMCCAP, May 2007
21
How to find services?

• Two complementary developments
• smaller devices carried on user instead of
  stationary devices
• devices that can be time-shared
• large plasma displays
• projector
• hi-res cameras
• echo-canceling speaker systems
• wide-area network access
• Need to discover services in local environment
• SLP (Service Location Protocol) allows querying
  for services
• find all color displays with at least XGA
  resolution
• slp//example.com/SrvRqst?public?typeprinter
• SLP in multicast mode
• SLP in DA mode
• Apple Bonjour
• Need to discover services before getting to
  environment
• is there a camera in the meeting room?
• SLP extension find remote DA via DNS SRV
• LoST to find services by geographic location

22
Session mobility
Local Devices
Transcoder
Internet
SLP DA
SLP SA
SLP UA
SIP SM
SIP UA
SIP UA
Correspondent Node (CN)
SLP SIP RTP
SIP SM
SIP UA
SLP UA
Mobile Node (MN)
23
Overview

• The original vision of ubiquitous computing
• User challenges
• Beyond terminal mobility
• Location as new core service
• Universality 7DS

24
Context-aware communication

• context the interrelated conditions in which
  something exists or occurs
• anything known about the participants in the
  (potential) communication relationship
• both at caller and callee

25
GEOPRIV and SIMPLE architectures
rule maker
DHCP
XCAP (rules)
target
location server
location recipient
notification interface
publication interface
GEOPRIV
SUBSCRIBE
presentity
presence agent
watcher
SIP presence
PUBLISH
NOTIFY
caller
callee
SIP call
INVITE
INVITE
26
Presence data architecture
presence sources
PUBLISH
raw presence document
privacy filtering
create view (compose)
depends on watcher
XCAP
XCAP
select best source resolve contradictions
composition policy
privacy policy
(not defined yet)
draft-ietf-simple-presence-data-model
27
Presence data architecture
candidate presence document
raw presence document
post-processing composition (merging)
watcher filter
SUBSCRIBE
remove data not of interest
difference to previous notification
final presence document
watcher
NOTIFY
28
Presence data model
calendar
cell
manual
person (presentity) (views)
alice_at_example.com audio, video, text
r42_at_example.com video
services
devices
29
RPID rich presence

• Provide watchers with better information about
  the what, where, how of presentities
• facilitate appropriate communications
• wait until end of meeting
• use text messaging instead of phone call
• make quick call before flight takes off
• designed to be derivable from calendar
  information
• or provided by sensors in the environment
• allow filtering by sphere the parts of our
  life
• dont show recreation details to colleagues

30
Rich presence

• More information for (authorized) people and
  applications
• automatically derived from
• sensors physical presence, movement
• electronic activity calendars
• Rich information
• multiple contacts per presentity
• device (cell, PDA, phone, )
• service (audio)
• activities, current and planned
• sphere (home vs. work)
• current user mood
• surroundings (noise, privacy, vehicle, )
• contact information
• composing (typing, recording audio/video IM, )

31
Presence and privacy

• All presence data, particularly location, is
  highly sensitive
• Basic location object (PIDF-LO) describes
• distribution (binary)
• retention duration
• Policy rules for more detailed access control
• who can subscribe to my presence
• who can see what when

lttuple id"sg89ae"gt ltstatusgt ltgpgeoprivgt
ltgplocation-infogt ltgmllocationgt
ltgmlPoint gmlid"point1 srsName"ep
sg4326"gt ltgmlcoordinatesgt374630N
1222510W lt/gmlcoordinatesgt
lt/gmlPointgt lt/gmllocationgt
lt/gplocation-infogt ltgpusage-rulesgt
ltgpretransmission-allowedgtno lt/gpretransmissi
on-allowedgt ltgpretention-expirygt2003-06-2
3T045729Z lt/gpretention-expirygt
lt/gpusage-rulesgt lt/gpgeoprivgt lt/statusgt
lttimestampgt2003-06-22T205729Zlt/timestampgt lt/tupl
egt
32
Events as missing Internet capability

• aka PUB/SUB
• Used across applications, e.g.,
• email and voicemail notification
• presence
• replace RSS ( poll!)
• web service completion
• emergency alerts (reverse 9-1-1)
• network management
• home control
• data synchronization
• Rich research history
• but too complex, optimize the wrong thing
• XMPP and SIP as likely transport candidates

33
Local Switch
Automatic Number Identification
Automatic Location Identification
Collaboration between local phone providers and
local public safety agencies
34
911 technology failures

• NY Times (An S O S for 911 Systems in Age of
  High-Tech), 4/6/07
• 40 of counties, most of them rural or
  small-town , cannot yet pinpoint the location of
  the cellphone callers, though the technology to
  do so has been available for at least five
  years.In Okmulgee, Okla., last November,
  4-year-old Graciella Mathews-Tiger died in a
  house fire after a 911 operator who lacked the
  technology to pinpoint the call misheard the
  address.
• Phase II wireless billions of dollars spent
• In Mississippi, only 1 of out 5 counties
• As it ages, it is cracking, with problems like
  system overload, understaffing, misrouted calls
  and bug-ridden databases leading to unanswered
  calls and dangerous errors.
• operator (CAMA) trunks, with 8-digit number
  delivery
• MSAG and ALI databases

35
Location delivery
GPS
HELD
HTTP
wire map
DHCP
LLDP-MED
36
Location, location, location, ...
Voice Service Provider (VSP) sees emergency
call but does not know caller location
ISP/IAP knows user location but does not handle
call
37
Options for location delivery

• Wireless
• GPS
• S5 wireless (active sensors triangulation)
• Skyhook (802.11 in urban areas)
• HDTV
• L2 LLDP-MED (standardized version of CDP
  location data)
• periodic per-port broadcast of configuration
  information
• L3 DHCP for
• geospatial (RFC 3825)
• civic (RFC 4676)
• L7 proposals for retrievals HELD, SIP,
• for own IP address or by third party (e.g., ISP
  to infrastructure provider)
• by IP address
• by MAC address
• by identifier (conveyed by DHCP or PPP)
• HTTP-based

38
Locating Caller using LLDP-MED
LLDP-MED stands for Link Layer Discovery
Protocol a vendor-neutral Layer 2 protocol
that allows a network device to advertise its
identity and capabilities on the local
network. Media Endpoint Discovery an
enhancement to the LLDP that allows discovery of
other things including location
From Wikipedia
I am LLDP-MED Capable. I can process location
information.
Your location is 500 W 120TH st. New York NY
10027
39
Location determination options
40
SIP message for Location Info.
------- _ZGY1NTFlZDJkMDkxY2FkMTIxMWI2MzIzNjE1M2U0
OTY MIME-Version 1.0 Content-Type
application/pidfxml Content-Transfer-Encoding
8bit lt?xml version"1.0" encoding"ISO-8859-1"?gt
ltpresence xmlns"urnietfparamsxmlnspidf"
xmlnsgp"urnietfparamsxmlnspidfgeopriv10"
xmlnscl" urnietfparamsxmlnspidfgeopriv1
0civilLoc" xmlnsgml"urnopengisspecificati
ongmlschema-xsdfeaturev3.0"
entity"sipcalltaker_ny2_at_irt.cs.columbia.edu"gt
lttuple id"28185"gt ltstatusgt ltgpgeoprivgt
ltgplocation-infogt ltclcivilAddressgt
ltclcountrygtuslt/clcountrygt
ltclA1gtnylt/clA1gt ltclA2gtnew
yorklt/clA2gt ltclA3gtnew yorklt/clA3gt
ltclA6gtamsterdamlt/clA6gt
ltclHNOgt1214lt/clHNOgt lt/clcivilAddressgt
lt/gplocation-infogt ltgpmethodgtManuallt/gp
methodgt lt/gpgeoprivgt lt/statusgt
ltcontact priority"0.8"gtsipeddie_at_160.39.54.70506
0lt/contactgt lttimestampgt2005-09-26T155734-040
0lt/timestampgt lt/tuplegt lt/presencegt -------
_ZGY1NTFlZDJkMDkxY2FkMTIxMWI2MzIzNjE1M2U0OTY--
INVITE urnservicesos SIP/2.0
request line
To urnservicesos Call-ID 763782461_at_192.168.1.1
06 Via SIP/2.0/TCP 192.168.1.1064064rport Conte
nt-Type multipart/mixed boundary From
sipcaller_at_irt.cs.columbia.edu Contact
ltsipeddie_at_160.39.54.705060gt CSeq 1
INVITE Content-Length 1379
header fields
------ _ZGY1NTFlZDJkMDkxY2FkMTIxMWI2MzIzNjE1M2U0O
TY MIME-Version 1.0 content-Type
application/sdp Content-Transfer-Encoding
8bit v0 oeddie 1127764654 1127764654 IN IP4
192.168.1.106 sSIPC Call cIN IP4
160.39.54.70 t0 0 maudio 10000 RTP/AVP 0
3 mvideo 20000 RTP 31
SDP
PIDF-LO
41
(No Transcript)
42
Tracking
43
Data formats location

• Civic (street)
• jurisdictional postal
• Geo (longitude latitude)
• point, polygon, circle,
• see GeoRSS for simple example

44
ECRIT LoST Functionality

• Civic as well as geospatial queries
• civic address validation
• Recursive and iterative resolution
• Fully distributed and hierarchical deployment
• can be split by any geographic or civic boundary
• same civic region can span multiple LoST servers

• Indicates errors in civic location data ?
  debugging
• but provides best-effort resolution
• Can be used for non-emergency services
• directory and information services
• pizza delivery services, towing companies,

ltfindService xmlns"urnlost1"gt ltlocation
profile"basic-civic"gt ltcivicAddressgt
ltcountrygtGermanylt/countrygt
ltA1gtBavarialt/A1gt ltA3gtMunichlt/A3gt
ltA6gtNeu Perlachlt/A6gt ltHNOgt96lt/HNOgt
lt/civicAddressgt lt/locationgt
ltservicegturnservicesos.policelt/servicegt lt/findSe
rvicegt
45
LoST Location-to-URL Mapping
VSP1
cluster serving VSP1
replicate root information
cluster serves VSP2
123 Broad Ave Leonia Bergen County NJ US
LoST
root nodes
NY US
NJ US
sippsap_at_leonianj.gov
search referral
Bergen County NJ US
Leonia NJ US
46
LoST Architecture
G
tree guide
G
G
G
broadcast (gossip)
T1 .us T2 .de
G
resolver
T2 (.de)
seeker 313 Westview Leonia, NJ US
T3 (.dk)
T1 (.us)
Leonia, NJ ? sippsap_at_leonianj.gov
47
Left to do event notification

• notify (small) group of users when something of
  interest happens
• presence change of communications state
• email, voicemail alerts
• environmental conditions
• vehicle status
• emergency alerts
• kludges
• HTTP with pending response
• inverse HTTP --gt doesnt work with NATs
• Lots of research (e.g., SIENA)
• IETF efforts starting
• SIP-based
• XMPP

48
Overview

• The original vision of ubiquitous computing
• User challenges
• Beyond terminal mobility
• Location as new core service
• Universality 7DS

49
Problems with Wide Area Wireless

• 802.11 currently hard to deploy across city or
  large area
• Problem How can mobile devices / gadgets get
  information while on the move?
• Use local peer-to-peer wireless networks to
  exchange information
• Peers can get information they do not have from
  another peer

Solution 7DS!
S. Srinivasan, A. Moghadam, S.G. Hong, H.
Schulzrinne, 7DS - Node Cooperation and
Information Exchange in Mostly Disconnected
Networks, ICC '07. June 2007.
50
How 7DS Works

• When devices are in the same BSS (Basic service
  set) of 802.11 ad-hoc network, they discover each
  other using service discovery of Zeroconf

zeroconf
51
How 7DS Works

• If there is no Internet connection, the devices
  can communicatewith each other to exchange
  information

Internet
52
Web Delivery Model

• 7DS core functionality Emulation of web content
  access and e-mail delivery

53
Design

• Peer-to-peer network set up using zeroconf
• Protocol enables devices to communicate with each
  other without a DHCP server, a DNS server and a
  Directory server
• Proxy server serves content
• Search engine searches for local data
• MTA store and forward
• In progress File synchronization, BBS

54
Store and Forward

• Forwarding e-mail in the ad-hoc network
• Acts as an MTA

55
Search Engine

• Provides ability to query self for results
• Searches the cache index using Swish-e library
• Presents results in any of three formats HTML,
  XML and plain text
• Similar in concept to Google Desktop

56
Query Multicast Engine

• Used to actually exchange information among peers
• Requesting peer broadcasts a query to the network
• Responding peers reply if they have information
• Send encoded string with list of matching items
• Requesting peer retrieves suitable information

57
File Synchronization
SERVICE ANNOUNCEMENT
SERVICE RESOLUTION
FILE SYNCHRONIZATION USING RSYNC PROTOCOL
SRV 7ds-fs1.filesync._7ds._udp.local.
7ds-device1.local2525 TXT file1.xml TXT
file2.xml
I want Word.doc and presentation.ppt
Word.doc Presentation.ppt
SRV 7ds-fs2.filesync._7ds._udp.local.
7ds-device2.local2525 TXT word.doc TXT
presentation.ppt
I want File1.xml and file2.xml
File1.xml File2.xml
File1.xml File2.xml
Word.doc Presentation.ppt
58
Conclusion

• Motivate mobile ubiquitous research by user
  problems
• From stovepipe mobile wireless systems to
  personal and shareable wireless networks
• Thinking beyond single applications
• presence ?event notification
• 9-1-1 location ? time location as
  infrastructure
• Need new models of creating services
• domain-specific languages, not Java APIs