Interprocess Communications in Distributed Operating Systems CS 239 Distributed Operating Systems April 9, 2001

1
Interprocess Communications in Distributed
Operating Systems CS 239Distributed Operating
SystemsApril 9, 2001

2
Introduction

• Message passing communications
• Message passing paradigms
• Pipes and sockets
• Multicast and broadcast
• Request/reply communications
• RPC
• Secure RPC

3
Message Passing Communications

• By definition, distributed systems require IPC
• Most distributed systems connected only by
  network
• So most communications must be by message
• Other IPC abstractions built on messages

4
Levels of Abstraction of Communication

Interprocess Communication
Transactions
Request/Reply (RPC)
Message Passing
Network OS
Transport Connection
Communications Network
Packet Switching
5
Basics of Messages

• Messages are collections of data objects
• Header
• Payload
• At message level, sender and receiver regarded as
  peers
• Neither is in complete control

6
Steps in Sending a Message

• Sending process formats message
• Invokes local OS to send it
• Local OS (usually) copies it into kernel space
• Local OS transfers message to local network
  interface
• Network interface passes message to network
  hardware

7
Steps in Receiving a Message

• Message delivered from network to receivers
  network interface
• Interface generates interrupt for OS
• OS copies message to buffer
• OS either interrupts receiver or waits for
  receive system call
• Message (usually) copied from OS buffer to
  receiver process

8
Moving the Message

Sender
Receiver
OS
OS
9
Basic Message Primitives

• send(destination, message)
• Transfer contents of message to location
  specified by destination
• receive(source,message)
• Accept a message from source and put it in
  location specified by message

10
What is a Source/Destination?

• Complete specification of identity of a remote
  process
• Typically, a process name
• Often including a machine name
• Often with some specification of an internal
  address within the process

11
Common Source/Destination Naming Options

• Process name
• Link
• Mailbox
• Port

12
Process Name

• Varies depending on system specifics
• Possible to have global, transparent process
  namespace
• More commonly, specify hosting node
• Is this a good idea if processes move?
• If only component of name, all messages to a
  process share one queue

13
Symmetric Process Naming

A
B
14
Assymmetry in Naming

• Sender assumed to know receivers address
• Sometimes, receivers dont know senders address
  ahead of time
• E.g., server handling multiple clients
• Sometimes receive() fills source into variable,
  rather than specifying value

15
Asymmetric Process Naming

A
C
B
16
Source/Destination Naming by Links

• Sometimes we want multiple communications paths
  between pairs of processes
• Connections
• We can name these connections as sources and
  destinations
• Links

17
Characteristics of Links

• Unidirectional
• Multiple links allowed between two processes
• Each link independently nameable
• Similar to multiple process entry points
• Dynamically created and destroyed
• Direct communications between peers

18
Links

create_link(B)
create_link(B)
create_link(A)
A
B
19
Source/Destination Naming by Mailboxes

• Links arent appropriate for all cases
• Sometimes sender doesnt care about identity of
  receiver
• E.g., obtaining service from one of pool of
  servers
• Or receiver doesnt care about identity of sender
• Mailboxes dont require matching one sender to
  one receiver

20
Mailbox Characteristics

• Multiple senders can put messages into the same
  mailbox
• Multiple receivers can retrieve messages from the
  same mailbox
• Varying semantics about how each of these works

21
Mailboxes
receive(mailbox,message)

Mailbox
receive(mailbox,message)
22
Message Synchronization and Buffering

• Sending and receiving a message requires some
  synchronization
• When is it OK to send?
• When is it OK to receive?
• When is the process complete?
• When can the system free various buffers?

23
Blocking and Non-blocking Send and Receiver

• Blocking send blocks the sender
• Blocking receive blocks the receiver
• For how long?
• Some options here
• Non-blocking versions allow send and receive to
  go ahead immediately

24
Blocking at the Receiver

• In most systems, receiver blocks
• Must wait for message to arrive before proceeding
• Sometimes, only blocks from moment of message
  arrival until message is in receiver buffer
• Sometimes from point when receiver needs message

25
Blocking on Arrival

Block!
A
B
26
Blocking on Demand

Block!
A
B
receive(A,message)
27
Common Options for Send Blocking

• Non-blocking send
• Blocking send
• Reliable blocking send
• Explicit blocking send
• Request and reply

28
Non-Blocking Send

Block!
29
Blocking Send

Block!
30
Reliable Blocking Send

Block!
ACK
31
Explicit Blocking Send

Block!
ACK
32
Request and Reply

Block!
33
Buffering

• Any delay in duration of blocking implies a
  buffering capability
• In the sending kernel
• In the network
• In the receiving kernel
• Buffering allows multiple simultaneous sends
• Also provides reliability

34
Pipes and Sockets

• A common OS features providing message
  communications
• The interface between applications and the
  network
• Hides many details of the network

35
Pipes

• Originally built as IPC for single processor
  system
• Basically a FIFO byte stream buffer
• Bytes fed in one end
• Bytes read out the other end
• In the same order

36
Pipe Details

• Once set up, treated like a file
• Typically, one process writes, another reads
• Usually parent and child processes
• Send is non-blocking
• Assuming local buffer space available
• Receive is blocking
• Receiver explicitly asks to receive bytes
• And waits for them to arrive

37
Pipes, Buffering, and Blocking

• Pipes are implemented in the kernel by setting up
  a buffer
• Of fixed size
• When writer fills the buffer, further writes
  block
• When reader asks for bytes from empty buffer,
  reads block

38
Named Pipes

• Normal pipes only usable by processes that share
  file descriptors
• Usually parents and children
• Pipes can have names attached
• In file system namespace
• Allows any process to name either end of pipe
• Implemented with kernel buffers or file system

39
Pipes and Distributed Processing

• Pipe implementations assume a shared buffer space
• Named pipes assume a shared namespace
• And shared FS or buffer space
• Generally, distributed systems might not have
  these
• Can we provide a similar mechanism for that
  environment?

40
Sockets

• A general mechanism for distributed systems IPC
• Subsumes pipes
• Sets up communication channel named by two unique
  endpoints

41
Creating Sockets

• Both communicating processes must issue socket()
  calls
• Creating a logical communications endpoint (LCE)
• Local to creating process
• Must bind LCE to a physical communications
  endpoint (PCE)

42
Example of Creating Sockets

sA socket(domain, type, protocol)
sB socket(domain, type, protocol)
domain describes protocol family to use
type describes high-level characteristics of
communications
protocol specifies which protocol in the family
to use
43
Binding Sockets

• A PCE is a network host address plus a transport
  port pair
• Network host address space is global
• Transport port pair numbers generated locally
• The bind() system call performs the bindings

44
Example of Binding Sockets

bind(sA,name,namelen)
bind(sB,name,namelen)
Now A can send a message to B using the socket
45
Connected Sockets

• Bound sockets can be used for connectionless
  message sending
• Sending process must know remote PCE, though
• Connecting sockets allows send calls without
  specifying remote PCE
• Typically, for servers with well-known ports

46
Broadcast and Multicast

• Methods for sending the same message to more than
  one destination
• Broadcast typically sends to everyone on the
  network
• Multicast typically sends to chosen set of
  recipients

47
Best Efforts Multicast

• Guarantees delivery to at least one member of the
  group
• Useful when trying to obtain service from set of
  interchangeable servers

48
Reliable Multicast

• Guarantees delivery to all servers
• Or none of them
• Useful for distributing shared information to
  cooperating processes
• If several reliable multicasts going on
  simultaneously, participants may see messages in
  different orders

49
Handling Failures in Multicast

• Sender expects some feedback from each recipient
• Perhaps explicit acknowledgement
• If not all received, re-multicast
• Or send to just those who didnt acknowledge
• Failure of sender harder to deal with

50
Multicast Message Ordering

• FIFO Messages from single source delivered in
  same order as sent
• Causal order Causally related messages from
  different sources delivered in causal order
• Total order All messages from all sources
  delivered in single order

51
Achieving FIFO Ordering

• Sender keeps local sequence counter
• Each multicast stamped with incremented counter
• Each receiver keeps counter for each sender
• Only deliver messages with expected timestamp
• Queue or reject others

52
Achieving Causal Order

• Use what amounts to vector clocks
• One element for each sender
• Sender increments local element of vector each
  time it multicasts
• Attaching new vector to message
• Each receiver stores vector of last message it
  accepted

53
Receiving Causally Ordered Messages Properly

• Compare incoming messages vector to stored
  vector
• If one element of incoming vector is 1 greater
  than stored vector, accept it
• Delay it if more than one element greater, or any
  element greater by more than 1
• Reject if vector is less than stored one

54
Example of Receiving Causally Ordered Messages
A

C
Should B accept this message?
But Bs message is lost
Should C accept this message?
No! 1 01, 0 0, and 1 01
Yes! 1 01, 0 0, and 0 0
B has missed a causal message
B
55
Totally Ordering Multicasts

• Even harder than causal order
• Essentially, requires a commitment protocol
• Everyone agrees theyve received the message
• The sender knows everyones agreed
• Then the message is delivered
• Related to 2-phase commit