Media Services in CINEMA

1
Media Services in CINEMA

• Presented by Kundan Singh
• Joint work with Henning Schulzrinne, Jonathan
  Lennox, Xiaotao Wu, Wenyu Jiang, Sankaran
  Narayanan,
• Ajay Nambi and Ali Khwaja
• Computer Science Department, Columbia University
• April 25, 2003

2
Purpose
N

• What are media services?
• What is CINEMA?
• What services are provided by CINEMA?
• Conferencing
• Streaming media
• Unified messaging
• Interactive voice response
• How can we use CINEMA?
• The discussion on services and specific CINEMA
  implementation components are intermixed.

Not relevant or not implemented On-going
work Yes, Implemented
N
O
Y
3
IP telephony services (PSTN)
O

• Call routing services pre-call, one party
• speed dial
• call forwarding
• follow me
• call filtering/blocking (in/out)
• do not disturb
• distinctive ringing
• Call handling features
• autoanswer

• Multi-party features
• call waiting
• call transfer (blind, consultative)
• conference call
• call park
• call pickup
• music on hold
• call monitoring

IP-telephony cant win by just providing what
PSTN has
4
IP telephony services (Internet)
O

• Presence-enabled calls
• place call only if callee is available
• Presence-enabled conferencing
• call conference participants when all are online
  and not busy
• IM conference alerts
• receive IM when someone joins a conference
• Unified messaging
• receive email, IM alert for new voicemails

5
Where the services reside?
O
Double ringing sound when boss calls
Enter your authentication PIN for billing
Use finger for locating user
B2BUA
Endpoint
Make call when boss is online
Proxy/registrar
Endpoint
Forward to office phone during day, and home
phone during evening

• Common gateway interface (CGI)
• Call processing language (CPL)
• SIP servlet

• Language for End System Services (LESS)

6
Bricks in media applications
N
Interactive voice response
Internet Telephony
Internet Radio/TV
Messaging and Presence
Unified messaging
Video conferencing
SIP
SAP
RSVP
RTCP
Media G.711 MPEG
RTP
RTSP
Quality of service
Media transport
Signaling
Program Call routing
Voice XML
DTMF
Mixing
Speech/ text
SDP
7
Audio Packet Transfer
N

• Digitization (e.g., sampling at 8kHz, 16 bits per
  sample, i.e, 128 kb/s or 320 bytes per 20 ms)
• Real-time compression/encoding (e.g., G.729A at 8
  kb/s)
• Transport to remote IP address and port number
  over UDP (Why not TCP?)
• Processing on receiver side is the reverse

8
Problems with UDP
N
Sender
1
2
3
4
5
6
7
(a)
(b)
1
2
3
5
7
6
timeline
Receiver

• Unreliable UDP
• Packet loss
• Out-of-order (very rarely)
• Jitter (delay variation)

9
Receive Buffer
Y

• Receive buffer to absorb jitter
• Tradeoff in buffer size
• Adaptive delay adjustment
• Sequence number to detect packet loss Just
  ignore the loss!

Sender
1
2
3
4
5
6
7
8
9
0
1
2
3
4
1
2
3
5
7
6
8
9
0
2
3
4
Receiver
1
2
7
1
2
5
7
8
9
0
2
3
1
2
3
5
6
7
8
9
0
2
10
Real-time Transport Protocol
Y
IP header
Sequence number
Payload type
CC
M
V
P
X
UDP header
Timestamp (proportional to sampling time)
RTP Header
Source identifier (SSrc)
Encoded Audio
Optional contributors list (CSrc)
msg
RTP media transport RTCP QoS feedback
sendto(, msg, ) recvfrom(, msg, )
11
Streaming media
Y

• Messages
• RTSP vs HTTP
• UDP, Stateful, Control, Timed
• Applications
• Recording, Playback, Live media, Conference

12
Columbia rtspd
Y

• Existing
• Playback/recording of ?-law (Sun .au files) and
  raw
• Unicast address
• Multi-threaded
• Portable (Unix/win32)
• Works with Apple QuickTime client
• UDP TCP
• IPv4 IPv6

• Planned
• Multicast
• MPEG video
• Mp3, wav audio
• Media over TCP
• Live media

13
RTP-based Conference
N
?-law
Mixer
Transcoder
?-law
?
?
G.729
G.729
?-law
?-law
14
Why do we need signaling?
Y

• What is the IP address of Alices host?
• What audio/video encoding can it support?

REGISTER as alicegt128.59.19.194
Call alice
128.59.19.194
Alices host 128.59.19.194
bobs host
home.com

• Session Initiation Protocol
• Email-like address alice_at_home.com
• DNS
• Proxy, redirect

15
Session description
Y
INVITE alice_at_home.com I can support ?-law and
G.729 Send me audio at 202.16.49.276780
Alice
Bob
128.59.19.194
202.16.49.27
OK I can support ?-law Send me audio at
128.59.19.1948000
RTP
To port 8000
RTP
To port 6780
16
Columbia sipc
Y
17
Emergency service
O

• Calling
• E911, location
• Assistance
• Audio/video, remote-help
• Notification
• Earthquake, fire

18
Presence/event notification
Y
office.com
bob_at_office.com
Presence server
PUA
PA
REGISTER
alice_at_home.com
SUBSCRIBE
NOTIFY
PUA
registrar
19
Columbia IM and presence
Y
20
Endpoint call control
O

• Language for End System Services (LESS) for
  endpoint service creation
• Direct user interaction, direct media control
• Handle converged information, e.g., call,
  presence, email
• Example when buddy is online, make a call
• ltless name"online_call"
• require"generic presence ui"gt
• ltnotification status"online" priority"0.5"gt
• ltaddress-switch field"origin"gt
• ltaddress isalice_at_office.com"gt
• ltcall /gt
• ltalert soundring.au"
• text"Calling " /gt
• lt/addressgt
• lt/address-switchgt
• lt/notificationgt
• lt/lessgt

21
Network call control
Y

• SIP-CGI
• CPL
• SIP servlets

if (defined ENVSIP_FROM ENVSIP_FROM
/sipboss_at_mycompany.com/) foreach reg
(get_regs()) print
"CGI-PROXY-REQUEST reg SIP/2.0\n" print
"Priority urgent\n\n"
22
Call transfer
O
B
A

• REFER
• Blind/ consultation/ attended

C
active call
REFER C Referred-By B
INVITE C Referred-By B
BYE A
active call
23
B2BUA and third-party call control
O

• Back-to-back UA
• Incoming call triggers outgoing call
• Services
• Calling card
• Anonymizer

B
A
C
INVITE
OK (SDP1)
ACK
INVITE (SDP1)
OK (SDP2)
ACK
INVITE (SDP2)
OK
ACK
24
Voicemail
Y
Endpoint redirects

• Problems in PSTN
• Design alternatives
• Issues

Redirect after 10s
alice_at_vmail.yahoo.com
Proxy controls
Voicemail acts like a phone
vmail.pl
SIP_FROM SIP_TO stdin
CGI-PROXY-REQUEST stdout
If no response
accept after 15s
25
Columbia sipum
Y

• Goals
• Universal access
• Scalability
• Provider independent
• Why SIP and RTSP?
• Reuse existing infrastructure and tools

26
Columbia sipum
Y

• Retrieval
• Web interface
• rtsp//server/alice /inbox/1677.au,
• sipalice-1677-retrieve_at_server
• press 1 to listen
• Configuration
• Folders
• Options
• Email

27
VoiceXML
Y
Gateway
sipvxml
PSTN
Telephone
IVR platform

• Voice and telephony functions
• (ASR, TTS, DTMF)
• Service logic (application specific)

28
VoiceXML contd.
Y
ltformgt ltfield nameidgt ltpromptgt
Your ID, please. lt/promptgt lt/fieldgt
ltblockgt ltsubmit nexturl/gt
lt/blockgt lt/formgt
ltform actionurlgt Enter your Id ltinput
nameidgt ltinput typesubmitgt lt/formgt
Telephony, speech synthesis or audio output, user
input and grammar, program flow, variable and
properties, error handling,
29
DTMF
Y
PSTN
Internet
Telephone
SIP/PSTN gateway
Internet user
Audio
D
SIP INFO
D
RTP/ RFC2833
30
Columbia sipvxml
Y
PSTN
Internet
Telephone
SIP/PSTN gateway

• Unified messaging access
• Email by phone
• Event notification and scheduling
• Audio volume control for conference
• Advanced conference control

TTS, ASR, DTMF, XML, HTTP, RTSP
sipvxml
SIP phone
Web server
Email.tcl
31
Email phone
Y
Email by phone
Inbox
Email
procmail
important mails
Internet
Email to IM

• Login
• Email formatting
• Listen, reply, delete, compose, forward
• Navigation -next, previous, jump

SIP

• Email formatting
• SIP based
• Text-to-speech

SIP
VoiceXML browser
TTS
SIP
HTTP
Internet
Email servlet
JSP
Inbox
DB
Email to phone
32
IM voice call
Y

• Who can initiate?
• authentication, billing,
• Feedback
• to voice user
• to IM user
• initial IM greeting
• Talk-spurt detection
• Speech recognition

SIP MESSAGE
TTS ASR
SIP INVITE
RTP
33
Unified messaging
O

• The gaps among different media (audio, video,
  text), devices (PC, phone) and means of
  communications (Email, SIP, IM) disappear for
  messaging

34
Notification
O

• Calendar
• Events
• Conferences

Schedule from a browser
SIP call 2129397063
IM kundan_at_
Web server
Calendar.cgi
Email Kns10_at_
at 600pm
35
Phone announcement server
Y
Text or audio
Destinations

• Input
• Range93970??
• List A, B, C
• Example
• Announcement
• Emergency
• Issues
• Voicemail
• Failure detection

TTS
PAS
SIP
. . .
36
RTSP TTS ASR
O
TTS ASR
Media server
SETUP rtsp//server/tts.cgi ?textHowareyou.
SETUP rtsp//server/asr.cgi
PLAY
RECORD
RTP
RTP
SET_PARAMETER TextI am fine, thank you.
37
Audio conference models (non-multicast)
N
Topology star
full-mesh
ad-hoc
No central point of failure
Advantages
Heterogeneous simple clients
Typically only three party conferences
Complex endpoints Complex signaling
Disadvantages
External server with high bandwidth link
38
Centralized conferencing
O

• Conference as URL
• staffmeet_at_office.com
• On the fly conferences
• my.adhoc_at_server
• Basic task join/leave
• Dial in, Refer dial in
• Dial out, Refer dial out

INVITE
INVITE
REFER
INVITE
server
REFER
39
Columbia sipconf
Y

• G.711, GSM, DVI, G.722 mixing (decode-mix-encode)
• Video replication IM text VNC screen sharing

40
Columbia sipconf
Y
41
Conference control
O

• Access control
• Floor control
• Notification
• State
• Participant active, departed, booted, failed
• dialog state, floor state
• Media
• sending received-by-all, muted
• receiving receiving-all, anchor-only
• ltconference-info version"0" state"full"
  entitymeeting_at_example.com"gt
• ltuser uri"sipalice_at_home.com"
  display-nameAlice"gt
• ltstatusgtactivelt/statusgt
• ltmedia-statusgt
• ltmedia-stream media-type"audio"/gt
• lt/media-statusgt
• lt/usergt
• ltuser uri"sipbob_at_office.com"
  display-nameBob Wilson"gt
• ltstatusgtdepartedlt/statusgt

42
Conference VoiceXML
Y
1. INVITE sipvxml
2. Call accepted
3. Enter your four digit PIN
4. Entered 4-6-8-3
5. Authenticate user, 4683gtAlice
6. Enter the conference identifier
7. Entered 2-3-
8. Permission to join, 23gtmeet
sipvxml
9. REFER meet_at_conference
10.Terminate the old call
Caller
11.INVITE meet_at_conference
Call transfer vs bridged mode
43
New conference applications
Y
The ease flexibility of sipvxml enables us to
build custom telephonic applications to suit our
needs. e.g., Volume Check Application
1. INVITE sipvxml
2. Menu 1. Vol Check 2. Mic Check
3. User enters 2
sipvxml
4. User speaks out a voice sample
5. Voice sample is analyzed
6. SipVXML Vol level too high/low/
Caller
7. User adjusts the vol level.
sipconf
8. User now joins conference.
44
Conferencing
Y

• Automatic volume adjustments
• Automatic load balancing
• Delay adjustments
• Conference recording
• Local or RTSP

Recording in a media server
45
Ncast MPEG distribution
Y
INVITE sipxyz_at_cs.columbia.edu
sip.cgi?sipxyz_at_cs.columbia.edu
46
Y
sipjohn_at_cs.columbia.edu
INVITE sipjohn_at_cs.columbia.edu
My owners SIP address is sipjohn_at_cs.columbia.edu
Help!!! (invoke sipc to call sipjohn_at_cs.columbia.
edu)
47
Summary of media services
Y

• Streaming media
• Playback, recording, live-media
• Unified messaging
• Voicemail, video/IM integration, ...
• Conferencing
• Mixing, load-balancing
• Interactive voice response
• Tele-banking, VoiceXML

48
CINEMAColumbia InterNet Extensible Multimedia
Architecture
Y
Distributed component architecture
User-agent (client) applications
CINEMA
Application and proxy servers
Libraries and APIs
Management and control
49
Columbia SIP user agent sipc
Y

• Media
• Audio, video, text, white board
• Screen sharing
• Shared web browsing
• Advanced
• Presence, instant messaging
• Programmable call handling
• Conference control (on-going)
• Emergency notification and handling (on-going)
• Device control
• Language for End System Services (on-going)

50
Columbia SIP servers (CINEMA)
Y
Single machine
Telephone switch
rtspd media server
Local/long distance 1-212-5551212
sipconf Conference server
Quicktime
RTSP
PSTN
RTSP clients
Department PBX
sipum Unified messaging
sipd Proxy, redirect, Registrar server
Internal Telephone Extn 7040
713x
SQL database
SIP/PSTN Gateway
Web based configuration
SNMP (Network Management)
H.323
siph323 SIP-H.323 translator
Video encoder
NetMeeting
51
Libraries and APIs
Y
CPL, CGI, Servlet
Parsing, SIP, SDP, RTP, mySQL interface, SNMP
interface, Portability stubs, etc.
52
Status of CINEMA services
Features/services Status
SIP proxy, redirect, registration server Product
SIP Presence agent programmable call routing Beta
Dial-in conference with audio, video, screen sharing, instant messaging, recording, load balancing, volume or delay adjustment, and IVR Beta
Dial-out or refer dial-in conferencing, conference control On-going
Voice mail with retrieval, deletion and IVR Beta
Media streaming server for basic audio record and play Beta
SIP/VXML browser for VoiceXML 1.0 with basic functionality Alpha
Call transfer, back-to-back user agent On-going
Phone announcement server, IM/call converter, email-by-phone and email-to-phone, SIP anonymizer Demo
Calendar event notification to IM, phone and email On-going
Web-based configuration and control Alpha
Media server with text-to-speech, speech-recognition, video, live media On-going
53
Whats next?from multimedia communications to
collaboration

• Synchronous communications
• Conferencing, IM
• Asynchronous communications
• Voicemails, message board, file sharing
• Ubiquitous computing
• i-button, ID-card
• Service creation
• User friendly, end-point/network

kns10_at_cs.columbia.edu http//www.cs.columbia.edu/
kns10/ http//www.cs.columbia.edu/IRT/cinema
54
Media Services in CINEMA

• BACKUP SLIDES

55
SIP beyond telephony

• SUBSCRIBE door_at_alice.home.net
• NOTIFY alice_at_work.com
• DO light_at_alice.home.net
• INVITE camera_at_alice.home.net

56
PSTN interworking
sipbob_at_home.com
1 212 9397063
Telephone network
Telephone subscriber
SIP/PSTN gateway
SIP server
IP endpoint

• Translating audio (PCMU/PCMA)
• Translating signaling (PRI/T1,ISUP)
• Overlap signaling
• Advanced features in SIP are lost in PSTN
• Translating identifiers (phone number)
• Determining transition points

57
PSTN to IP

• Gateway knows the SIP server
• ltsip7063_at_conductor.cs.columbia.eduuserphonegt
• ENUM
• DNS
• 1 212 9397042 gt 2.4.0.7.9.3.9.2.1.2.1.e164.arpa
  gt siphgs_at_cs.columbia.edu
• Suitable for relatively static contacts

58
IP to PSTN

• Static mapping
• 1-212-939xxxx gt _at_itgw1.cs.columbia.edu
• ITGW information is dynamic
• Overlapping networks
• Multiple providers
• Load balancing
• TRIP
• Route advertisement
• Can be implemented in outbound proxy
• Suitable for current hierarchical network
• 1 _at_service.mci.com at 4/min
• 1212 _at_nyc.gw.com at 1/min
• 1212939 _at_itgw1.columbia.edu free

59
SIP-CGI

• Programming language independent
• Maintains state via an opaque token
• For SIP proxies and endpoints
• call routing
• controlling forking
• call rejection
• call modification (Priority, Call-Info)
• RFC 3050
• Upload via web or REGISTER

60
SIP CGI contd.

• Block _at_anonymous.com
• if (defined ENVSIP_FROM
• ENVSIP_FROM "sip_at_anonymous.com")
• print "SIP/2.0 600 I can't talk right
  now\n\n"
• Make calls from boss as urgent
• if (defined ENVSIP_FROM
• ENVSIP_FROM /sipboss_at_mycompany.com/)
• foreach reg (get_regs())
• print "CGI-PROXY-REQUEST reg SIP/2.0\n"
• print "Priority urgent\n\n"

61
CPL

• XML-based interpreted
• Control
• Restricted to branching and subroutines
• no variables, no loops
• structured tree describing actions performed on
  call setup event
• Event
• incoming and outgoing events
• Switch
• address, language, time, priority
• Action
• Proxy, redirect, reject (busy)
• Graphical tools

• Protocol-independent
• SIP, H.323
• Extensions
• CPL for presence
• Logging
• Email

62
CPL contd.

• lt?xml version"1.0" ?gt
• lt!DOCTYPE cpl PUBLIC "-//IETF//DTD RFC2824 CPL
  1.0//EN" "cpl.dtd"gt
• ltcplgt
• ltincominggt
• lttime-switchgt
• lttime dtstart"20020830T140000Z"
• dtend"20020830T150000Z"gt
• ltreject status"reject" reasonBusy" /gt
• lt/timegt
• lt/time-switchgt
• lt/incominggt
• lt/cplgt

63
SIP servlet

• Java-based
• Runs in SIP server
• Receives SIP objects and processes them

• public class RejectServlet extends
  SipServletAdapter
• public boolean doInvite(SipRequest req)
• SipResponse res req.createResponse()
• res.setStatus(603)
• res.send()
• return true